Scraper¶
Scraper is the core app used to scrape the websites, parse the information and save it in its respective models.
Scraper admin¶
Make the posts modifiables in the admin
Just add the following code to add a simple model view:
from scraper.models import Post
admin.site.register(Post)
You can also check the following link for a better approach: https://docs.djangoproject.com/en/dev/ref/contrib/admin/
Scraper models¶
We are gonna find the Post model in here which is basically every post in the app the goal is to have a new post
for every new news on each news platform
post_type
is the way we can differentiate between news platforms posts
Every other field is well self-explanatory.
Scraper serializers¶
The PostSerializer
is responsable to serialize the Post
model.
By now this gonna return every field.
Scraper services¶
We have the ScraperManager
this is the core of the hole project due to is its responsability to scrape every news platform looking for the most recent post and save it into our DB
Scraper tests¶
TODO
Scraper views¶
PostViewSet
will handle our posts
methods logic, it haves all the methods inherited from ModelViewSet
.
Scraper commands¶
scrape
command is gonna be used to scrape every post on the news platforms using the ScraperManager
is gonna be used as a cronjob