The Concept of Dynamic Affiliate Product Recommendations

What if I could set up a page with affiliate links and make them automatically adjust based on price, inventory level, and commission rates? Sounds like an interesting idea, right? The concept is to build an app that can track the prices of products on multiple sites offering affiliate commissions. This way, when someone visits the page, they would see the product with the best deal, whether it's in stock or not.

Building such an App Would Require Significant Development

To make this a reality, I would need to build web scrapers to extract data from each site. The problem is that every single product and site might require separate web scrapers. Additionally, I would need to fetch the affiliate commission rates from different sites, which could be another hurdle. When scaling up, I'm sure I'd encounter issues with frequency-based updates. That's why I think tools like Bright Data could help solve these problems.

Storing and Updating Product Data

Once I have all this data, I'd need to store it in a database. It would be essential to track history and automatically adjust the recommendations. The user interface could either be custom-built for my own site or designed as a Shopify app, allowing me to integrate with any store's platform.

Exploring the Potential of E-commerce

This idea might seem daunting, but I believe it has potential. E-commerce is an area where businesses can create significant value, and developers who think creatively about how code impacts the bottom line are in high demand. Even if this project doesn't reach production, showcasing such a well-thought-out concept on my resume would be beneficial.

Building a Project like This Can Be a Learning Experience

Developing a web scraper to track product prices across multiple sites requires understanding web development and scraping techniques. I'm sure that this type of project can help developers gain experience in these areas. Moreover, building an app with dynamic affiliate recommendations demonstrates the ability to think about business needs and how code impacts those needs.

The Importance of Considering Business Needs

In today's e-commerce landscape, companies are looking for developers who not only know how to code but also understand how their code can drive value for businesses. Being able to develop projects that address real-world problems or enhance existing experiences is essential in this industry.

"WEBVTTKind: captionsLanguage: enI'm going to share with you three incredible opportunities for developers in the e-commerce space now these are projects that would look fantastic on your resume and if you decided to build all the way to production could actually make you a ton of money and be a million dooll project idea now the e-commerce space is one of the largest Industries in the world and most of the big players are actually tech companies I think Shopify Amazon Walmart Etc now I know this because I flew all the way to Las Vegas to and Shop talk 2024 when I was there I talked with some of the largest companies and I asked them what kind of developer Talent they were looking for now the biggest topic that kept popping up was data and the e-commerce space data is so important and these companies told me that they're constantly looking for developers that know how to collect data manage it and analyze it so if you can demonstrate those skills you're really going to stand out and that's what's inspired this video so with that said let's get into them now for each of the projects I share with you I'm going to break down what the idea is and then I'm going to give you how I would go about building this project but truthfully for all of these projects you're going to need a lot of data specifically web data which means you're going to need to build out some web scrapers now there's a lot of different ways to do this but when you're building up these web scrapers if you want to scale them up and actually collect hundreds or thousands of pages of data you're probably going to run into a lot of issues for example you may have captures that are blocking you you may get rate limited you may have your IP address banned if you hit a website too many times and that's where the sponsor of this video bright data comes in now I've worked with bright data for over 3 years and genuinely they're the only tool that I know that can solve all of these issues bright data provides a bunch of different tools from their web unlocker to their scraping browser which can automatically unblock websites for you and allow you to do web scraping at scale I've done a bunch of projects on my channel where I've showed you how to use those I'll link some on the screen right now but the idea is you can actually scale out you can use their automated browsers and you can control them from your local machine with Frameworks that you already know like selenium playwright and Puppeteer now I'll talk more about them in this video when we discuss how to build the web scrapers to collect this data but if you are trying to do web scraping at scale definitely check them out from the link in the description so with that said let's dive in the first project so the first thing I would consider building is a competitive price monitoring and optimization system now in the e-commerce field prices are constantly changing companies are running promotions items are going out of stock and it would shock you how many times the same product is available for a different price across different websites obviously as a business having access that information would be extremely important and allow you to adjust your prices based on it so the idea here is to build some kind of tool that can continuously collect this data so maybe you're going to track one specific product or a few different products that are competing across different sites you're going to track what the current price is you can also track information like the inventory level promotions Etc this way you can build some kind of dashboard and you can display all of the different prices for this product and give a business the ability to dynamically adjust their pricing based on the prices of competing products now you could build out some tool that automatically does the adjustment or applies or recommend some kind of price but at minimum having access to that information having a dashboard that shows it and being able to update in real time would be absolutely invaluable now I'm not going to say this is easy to build out but this would be something that would look fantastic on a resume and if you actually built it all the way out and made sure it could scale this would be something you could sell especially to maybe smaller businesses that don't have access to these huge tools and make a ton of money from so let me break down how you can go about building this project to kind of give you a head start if you do want to try it out so for this project you'll have to build out a few different things but where I would start is by building the web scrapers now I say scrapers because you're going to need to build a different one for every site that you want to scrape so maybe there's three or four targets that you want to look at maybe you want to look at Walmart Amazon Wayfair New Egg whatever you want to look at you're going to have to build out a scraper for all of those different sites and you want to build one that's Dynamic that can take in some kind of search string or some kind of text and then go to that site and find items that match that filter that way you don't need to build a different scraper for every different product you just build one that works Works generally for a specific site and then you can make it so it can scrape all the different pages or it can just scrape maybe the first product that appears in that result there's a bunch of different ways to set this up but the idea is you want to build a scraper that can go to a site and collect data on individual products now how do you do that well there's a few different ways uh I would personally use a framework like selenium but you could also use something like playwright or Puppeteer where you're actually going to automate the browser you might need to control it by maybe searching something rendering some Dynamic content and then you have to go through the Dom so the document object model all the HTML and search for specific Target elements and extract the data you want from it now in development when you're testing out you can probably just do this from your local machine but if you do want to scale this up or you want to be hitting the page maybe every five or 10 seconds you are going to need to push this out to some kind of infrastructure and this is where I would recommend using something like bright data that make it super easy just to spin up a remote browser that browser can do the scraping and then give the results back to you and you can have as many instan of them running as you want so rather than being limited to the few instances that you could run on your local machine you could literally spin up 10,000 bright data browsers that can go collect the data and they'll automatically bypass all of the blocks that you'll probably run into locally like captas um IP bands all the different things that you're bound to see if you do actually try to build out a project like that now once we have the scrapers built out and we can collect data on different products we need to store that data so we're probably going to dump it in some kind of database and then we'll want to build a dashboard so probably a nice front end build it in whatever you're comfortable with where we can see maybe a history of the prices that'd be useful especially if we're collecting that data anyways and then we'd want maybe a comparison of the same product on different uh websites right so just a nice table where we could see those prices from there you could go a step further and integrate this into a system where it automatically updates prices but I would probably just make it so it recommends a price for your product based on the prices of the other product now beyond that there's a lot more that you could do but what you're going to want to set up here is some kind of automation where you're running your browsers every few minutes or every hour or Whatever frequency you want so you're getting relatively upto-date data it's not going to be possible to run it like every millisecond or every two seconds but maybe you want to run it every 12 hours every 6 hours whatever time period you think is reasonable it will go out scrape the data collect it and then show you that real upto-date information now my next idea would be to build a customer sentiment and satisfaction analysis platform the idea here is rather than looking at prices we want to look at reviews and ratings this way we can actually analyze the sentiment of them using natural language processing and we can figure out which products people like and which they don't like we can also take all of these reviews we could actually run them into something like an llm and we could ask the llm to give us what the general opinion and feedback is we could actually get a written response on maybe improvements we could make to this product to make it better to increase the sales get better reviews Etc so this tool would be great because it could do two things for us it can tell us where there's a gap in the market and where we could make a new product that would fill that Gap or we could run it on our own products and we could see which ones are performing well which ones are poor and how we can improve existing ones that we already have so there's a lot of different use cases here but this is unique because we're getting outside of price and we're really doing kind of a mixture of data collection and then sentiment analysis which really is pretty impressive and if you have this on a resume it gives you a lot to talk about and it's something that really does fill a real business need so let's talk about how you could build out a project like this now I could just do something basic where I get if it's a positive sentiment or a negative sentiment what I think would be really interesting is if I actually took all of the different comments or the reviews and I passed it to a model something like chat GPT or something that's running locally on my computer and I asked it to summarize what the overall sentiment was this way I can actually get some more actionable um kind of feedback or I can get some string of text that tells me generally what people are thinking about this product so rather than just seeing that it's a three out of five or that people think it's good or bad I can see what people are mostly saying about it so maybe there's one defect and chat gbt kind of finds that and summarizes it for me I think that'd be a really interesting way to go about solving this problem anyways once I have the sentiment or I have that summary of what people are saying about the product what I would want to do obviously store that so I need some kind of database I'd probably use something like mongod DB and then I'm going to build some kind of dashboard that would allow me to view all of this information I think the really cool component here would be to get that kind of summary of what people generally are saying or thinking about the product that's going to give you a lot more insight than just a review number or manually having to go through and read all of the different reviews so that's project idea number two let's get into the third one so that leads me to my last idea which is definitely the most difficult but also the one I think has real potential and could make a ton of money now this is a crossplatform product tracker and affiliate aggregator now let me just give you an example of where you would use this so you can see kind of how it works and what the benefit would be so let's say that I'm a YouTuber right as I am a lot of you guys ask me what's your microphone what's your keyboard you want to know my gear so naturally what I would do is set up a page that has some different affiliate links on it and I would just probably go and use something like Amazon cuz it's the easiest one for me to navigate now the issue is you might go to that page and when you click that link the product might be out of stock or I might not be getting any affiliate commission from it or it might be a really high price compared to somewhere else you're just not going to buy it there so what I was thinking is what if I could build some kind of app that could plug into some store that could track the price of the same product on a few different sites that all offer some affiliate commission then what I could do is I could figure out okay what's the price of this product is it in stock and what's the commission that I'm going to get if you were to purchase from that link I then could programmatically and dynamically adjust the product that I'm showing you to make sure first of all there's always one that's in stock so if the Amazon one goes out of stock then I show you the one that's on Walmart instead or alternatively if I wanted to be greedy I could always just show you the one that has the highest commission for me so if if Walmart's offering me 6% and Amazon's only offering me 5% well I'm going to show the Walmart one as opposed to the Amazon one because I don't really care where you buy it from right and then same thing if I wanted to be nice I could always just show you the one that has the lowest price so that you guys are always getting the best deal I think that would be really really cool I think it'd be really complicated to build out because what I need to do is track not only the price inventory level and the commission rate but then I need to dynamically adjust that on the site and I need to make sure this works for really any type of product so I don't know exactly how you would go about building this out my intuition would be okay we need to build a web scraper in fact we're probably going to need to build two for every single product or not every single product every single site we're tracking because I need to build a web scraper for the product but then I need to go and look at what the affiliate commission rate is which is probably from a different site so I need to do that for all the different sites that I'm possibly going to show affiliate links for and then I need to write something that's going to do that automatic update based on whatever preference I would be setting now I'll tell you that when you do try to build these type of web scrapers I guarantee you you will run into issues if you are scaling this up especially if you're triggering them to run run every 10 minutes 20 minutes or like a high frequency if that's the case I do highly recommend you check out bright data they have all kinds of tools that really solve all of this for you and they make what seems like a really daunting task actually doable because whatever you do in development you can just push it out to their infrastructure and it's really just handled for you again I have a bunch of videos on my channel where I show you projects using that but I recommend you check it out if you do want to attempt something that's this complicated anyways once I do that I'm now obviously going to need to store all of that in a database I probably want to have the history and track it I need to automatically adjust my site and that's pretty much it so really the front end or the user interface would depend on where I'm showing this I could write something custom for my own site or I could build something like a Shopify app where I could just plug into anyone's store and now I can be the one taking affiliate commissions or I could sell them that as a gassas um or some kind of tool right anyways I think this is a really really cool idea again pretty complicated to build out but maybe it gave you some inspiration or kind of got the wheels spinning which is really the point of these type of videos I hope you got some inspiration from this I hope you're thinking about these opportunities e-commerce really is one of those spaces where you can do quite a lot and you can really drive a lot of value for businesses and even if you don't build something that goes to production these projects look really really good on a resume and they show that you are actually thinking about business needs which a lot of companies aren looking for that kind of well-rounded developer that can code but also understands how their code impacts the bottom line I'll wrap it up here here if you guys enjoyed leave a like subscribe and I will see you in the next oneI'm going to share with you three incredible opportunities for developers in the e-commerce space now these are projects that would look fantastic on your resume and if you decided to build all the way to production could actually make you a ton of money and be a million dooll project idea now the e-commerce space is one of the largest Industries in the world and most of the big players are actually tech companies I think Shopify Amazon Walmart Etc now I know this because I flew all the way to Las Vegas to and Shop talk 2024 when I was there I talked with some of the largest companies and I asked them what kind of developer Talent they were looking for now the biggest topic that kept popping up was data and the e-commerce space data is so important and these companies told me that they're constantly looking for developers that know how to collect data manage it and analyze it so if you can demonstrate those skills you're really going to stand out and that's what's inspired this video so with that said let's get into them now for each of the projects I share with you I'm going to break down what the idea is and then I'm going to give you how I would go about building this project but truthfully for all of these projects you're going to need a lot of data specifically web data which means you're going to need to build out some web scrapers now there's a lot of different ways to do this but when you're building up these web scrapers if you want to scale them up and actually collect hundreds or thousands of pages of data you're probably going to run into a lot of issues for example you may have captures that are blocking you you may get rate limited you may have your IP address banned if you hit a website too many times and that's where the sponsor of this video bright data comes in now I've worked with bright data for over 3 years and genuinely they're the only tool that I know that can solve all of these issues bright data provides a bunch of different tools from their web unlocker to their scraping browser which can automatically unblock websites for you and allow you to do web scraping at scale I've done a bunch of projects on my channel where I've showed you how to use those I'll link some on the screen right now but the idea is you can actually scale out you can use their automated browsers and you can control them from your local machine with Frameworks that you already know like selenium playwright and Puppeteer now I'll talk more about them in this video when we discuss how to build the web scrapers to collect this data but if you are trying to do web scraping at scale definitely check them out from the link in the description so with that said let's dive in the first project so the first thing I would consider building is a competitive price monitoring and optimization system now in the e-commerce field prices are constantly changing companies are running promotions items are going out of stock and it would shock you how many times the same product is available for a different price across different websites obviously as a business having access that information would be extremely important and allow you to adjust your prices based on it so the idea here is to build some kind of tool that can continuously collect this data so maybe you're going to track one specific product or a few different products that are competing across different sites you're going to track what the current price is you can also track information like the inventory level promotions Etc this way you can build some kind of dashboard and you can display all of the different prices for this product and give a business the ability to dynamically adjust their pricing based on the prices of competing products now you could build out some tool that automatically does the adjustment or applies or recommend some kind of price but at minimum having access to that information having a dashboard that shows it and being able to update in real time would be absolutely invaluable now I'm not going to say this is easy to build out but this would be something that would look fantastic on a resume and if you actually built it all the way out and made sure it could scale this would be something you could sell especially to maybe smaller businesses that don't have access to these huge tools and make a ton of money from so let me break down how you can go about building this project to kind of give you a head start if you do want to try it out so for this project you'll have to build out a few different things but where I would start is by building the web scrapers now I say scrapers because you're going to need to build a different one for every site that you want to scrape so maybe there's three or four targets that you want to look at maybe you want to look at Walmart Amazon Wayfair New Egg whatever you want to look at you're going to have to build out a scraper for all of those different sites and you want to build one that's Dynamic that can take in some kind of search string or some kind of text and then go to that site and find items that match that filter that way you don't need to build a different scraper for every different product you just build one that works Works generally for a specific site and then you can make it so it can scrape all the different pages or it can just scrape maybe the first product that appears in that result there's a bunch of different ways to set this up but the idea is you want to build a scraper that can go to a site and collect data on individual products now how do you do that well there's a few different ways uh I would personally use a framework like selenium but you could also use something like playwright or Puppeteer where you're actually going to automate the browser you might need to control it by maybe searching something rendering some Dynamic content and then you have to go through the Dom so the document object model all the HTML and search for specific Target elements and extract the data you want from it now in development when you're testing out you can probably just do this from your local machine but if you do want to scale this up or you want to be hitting the page maybe every five or 10 seconds you are going to need to push this out to some kind of infrastructure and this is where I would recommend using something like bright data that make it super easy just to spin up a remote browser that browser can do the scraping and then give the results back to you and you can have as many instan of them running as you want so rather than being limited to the few instances that you could run on your local machine you could literally spin up 10,000 bright data browsers that can go collect the data and they'll automatically bypass all of the blocks that you'll probably run into locally like captas um IP bands all the different things that you're bound to see if you do actually try to build out a project like that now once we have the scrapers built out and we can collect data on different products we need to store that data so we're probably going to dump it in some kind of database and then we'll want to build a dashboard so probably a nice front end build it in whatever you're comfortable with where we can see maybe a history of the prices that'd be useful especially if we're collecting that data anyways and then we'd want maybe a comparison of the same product on different uh websites right so just a nice table where we could see those prices from there you could go a step further and integrate this into a system where it automatically updates prices but I would probably just make it so it recommends a price for your product based on the prices of the other product now beyond that there's a lot more that you could do but what you're going to want to set up here is some kind of automation where you're running your browsers every few minutes or every hour or Whatever frequency you want so you're getting relatively upto-date data it's not going to be possible to run it like every millisecond or every two seconds but maybe you want to run it every 12 hours every 6 hours whatever time period you think is reasonable it will go out scrape the data collect it and then show you that real upto-date information now my next idea would be to build a customer sentiment and satisfaction analysis platform the idea here is rather than looking at prices we want to look at reviews and ratings this way we can actually analyze the sentiment of them using natural language processing and we can figure out which products people like and which they don't like we can also take all of these reviews we could actually run them into something like an llm and we could ask the llm to give us what the general opinion and feedback is we could actually get a written response on maybe improvements we could make to this product to make it better to increase the sales get better reviews Etc so this tool would be great because it could do two things for us it can tell us where there's a gap in the market and where we could make a new product that would fill that Gap or we could run it on our own products and we could see which ones are performing well which ones are poor and how we can improve existing ones that we already have so there's a lot of different use cases here but this is unique because we're getting outside of price and we're really doing kind of a mixture of data collection and then sentiment analysis which really is pretty impressive and if you have this on a resume it gives you a lot to talk about and it's something that really does fill a real business need so let's talk about how you could build out a project like this now I could just do something basic where I get if it's a positive sentiment or a negative sentiment what I think would be really interesting is if I actually took all of the different comments or the reviews and I passed it to a model something like chat GPT or something that's running locally on my computer and I asked it to summarize what the overall sentiment was this way I can actually get some more actionable um kind of feedback or I can get some string of text that tells me generally what people are thinking about this product so rather than just seeing that it's a three out of five or that people think it's good or bad I can see what people are mostly saying about it so maybe there's one defect and chat gbt kind of finds that and summarizes it for me I think that'd be a really interesting way to go about solving this problem anyways once I have the sentiment or I have that summary of what people are saying about the product what I would want to do obviously store that so I need some kind of database I'd probably use something like mongod DB and then I'm going to build some kind of dashboard that would allow me to view all of this information I think the really cool component here would be to get that kind of summary of what people generally are saying or thinking about the product that's going to give you a lot more insight than just a review number or manually having to go through and read all of the different reviews so that's project idea number two let's get into the third one so that leads me to my last idea which is definitely the most difficult but also the one I think has real potential and could make a ton of money now this is a crossplatform product tracker and affiliate aggregator now let me just give you an example of where you would use this so you can see kind of how it works and what the benefit would be so let's say that I'm a YouTuber right as I am a lot of you guys ask me what's your microphone what's your keyboard you want to know my gear so naturally what I would do is set up a page that has some different affiliate links on it and I would just probably go and use something like Amazon cuz it's the easiest one for me to navigate now the issue is you might go to that page and when you click that link the product might be out of stock or I might not be getting any affiliate commission from it or it might be a really high price compared to somewhere else you're just not going to buy it there so what I was thinking is what if I could build some kind of app that could plug into some store that could track the price of the same product on a few different sites that all offer some affiliate commission then what I could do is I could figure out okay what's the price of this product is it in stock and what's the commission that I'm going to get if you were to purchase from that link I then could programmatically and dynamically adjust the product that I'm showing you to make sure first of all there's always one that's in stock so if the Amazon one goes out of stock then I show you the one that's on Walmart instead or alternatively if I wanted to be greedy I could always just show you the one that has the highest commission for me so if if Walmart's offering me 6% and Amazon's only offering me 5% well I'm going to show the Walmart one as opposed to the Amazon one because I don't really care where you buy it from right and then same thing if I wanted to be nice I could always just show you the one that has the lowest price so that you guys are always getting the best deal I think that would be really really cool I think it'd be really complicated to build out because what I need to do is track not only the price inventory level and the commission rate but then I need to dynamically adjust that on the site and I need to make sure this works for really any type of product so I don't know exactly how you would go about building this out my intuition would be okay we need to build a web scraper in fact we're probably going to need to build two for every single product or not every single product every single site we're tracking because I need to build a web scraper for the product but then I need to go and look at what the affiliate commission rate is which is probably from a different site so I need to do that for all the different sites that I'm possibly going to show affiliate links for and then I need to write something that's going to do that automatic update based on whatever preference I would be setting now I'll tell you that when you do try to build these type of web scrapers I guarantee you you will run into issues if you are scaling this up especially if you're triggering them to run run every 10 minutes 20 minutes or like a high frequency if that's the case I do highly recommend you check out bright data they have all kinds of tools that really solve all of this for you and they make what seems like a really daunting task actually doable because whatever you do in development you can just push it out to their infrastructure and it's really just handled for you again I have a bunch of videos on my channel where I show you projects using that but I recommend you check it out if you do want to attempt something that's this complicated anyways once I do that I'm now obviously going to need to store all of that in a database I probably want to have the history and track it I need to automatically adjust my site and that's pretty much it so really the front end or the user interface would depend on where I'm showing this I could write something custom for my own site or I could build something like a Shopify app where I could just plug into anyone's store and now I can be the one taking affiliate commissions or I could sell them that as a gassas um or some kind of tool right anyways I think this is a really really cool idea again pretty complicated to build out but maybe it gave you some inspiration or kind of got the wheels spinning which is really the point of these type of videos I hope you got some inspiration from this I hope you're thinking about these opportunities e-commerce really is one of those spaces where you can do quite a lot and you can really drive a lot of value for businesses and even if you don't build something that goes to production these projects look really really good on a resume and they show that you are actually thinking about business needs which a lot of companies aren looking for that kind of well-rounded developer that can code but also understands how their code impacts the bottom line I'll wrap it up here here if you guys enjoyed leave a like subscribe and I will see you in the next one\n"