Sunday, 9 March 2014

Disqus Comments

XML watcher has been running well all day, sending me a report back every half an hour. It looks at about 1400 URLs each time (taken from 133 RSS) feeds and anything between 0 and 20 of these are new. Usually only a couple of new URLs every half hour, but I expect that there are fewer new articles due to it being weekend.

Found out how to bypass AJAX and load the full JSON of disqus comments via direct URL. Began working on loading and parsing these, but still need to put everything together into a crawler. Using Beautiful Soup to load the JSON section of the HTML page.

Need to find out how long the comment threads stay open for. Then we can easily go through the DB and crawl the comments for all the closed threads. (Or we can do this on a regular basis and check each to see if it's closed).

What information is important? An example disqus comment looks as follows (lots of metadata. Most of it could be useful, though can probably throw out at least the avatar stuff. Also, why raw_message and message - how do they differ; which should we keep?)

Installed browser automation tool, Selenium, in case we need to do anything else AJAX related.

{"isFlagged":false,"forum":"iol","parent":null,"author":{"username":"HARRYHAT1950","about":"","name":"HARRYHAT1950","url":"","isAnonymous":false,"rep":1.345875,"profileUrl":"http://disqus.com/HARRYHAT1950/","reputation":1.345875,"location":"","isPrivate":false,"isPrimary":true,"joinedAt":"2012-04-23T10:23:08","id":"25208750","avatar":{"small":{"permalink":"//a.disquscdn.com/uploads/forums/128/5645/avatar32.jpg?1386858689","cache":"//a.disquscdn.com/uploads/forums/128/5645/avatar32.jpg?1386858689"},"large":{"permalink":"//a.disquscdn.com/uploads/forums/128/5645/avatar92.jpg?1386858689","cache":"//a.disquscdn.com/uploads/forums/128/5645/avatar92.jpg?1386858689"},"permalink":"//a.disquscdn.com/uploads/forums/128/5645/avatar92.jpg?1386858689","cache":"//a.disquscdn.com/uploads/forums/128/5645/avatar92.jpg?1386858689"}},"media":[],"isDeleted":false,"isApproved":true,"dislikes":0,"raw_message":"Point 2: \"then your first call should be to an ambulance service or the traffic department who will, in turn, alert them\". This was obviously not written in SA. The first thing the traffic department does is notify their contact at their chosen towing company and negotiate their commission. After that they contact their favourite paramedic and negotiate their commission from them. Even the cops don't notify the provincial ambulance service '\u00e7os they know there is no commission and the provincial ambulance service is useless and won't turn up anyway.","createdAt":"2014-03-08T07:26:59","id":"1276210870","thread":"2379235034","depth":0,"numReports":0,"likes":8,"isEdited":false,"message":"\u003cp>Point 2: \"then your first call should be to an ambulance service or the traffic department who will, in turn, alert them\". This was obviously not written in SA. The first thing the traffic department does is notify their contact at their chosen towing company and negotiate their commission. After that they contact their favourite paramedic and negotiate their commission from them. Even the cops don't notify the provincial ambulance service '\u00e7os they know there is no commission and the provincial ambulance service is useless and won't turn up anyway.\u003c/p>","isSpam":false,"isHighlighted":false,"points":8}


12 comments:

  1. toptan iç giyim tercih etmenizin sebebi kaliteyi ucuza satın alabilmektir. Ürünler yine orjinaldir ve size sorun yaşatmaz. Yine de bilinen tekstil markalarını tercih etmelisiniz.

    Digitürk başvuru güncel adresine hoşgeldiniz. Hemen başvuru yaparsanız anında kurulum yapmaktayız.

    tutku iç giyim Türkiye'nin önde gelen iç giyim markalarından birisi olmasının yanı sıra en çok satan markalardan birisidir. Ürünleri hem çok kalitelidir hem de pamuk kullanımı daha fazladır.

    nbb sütyen hem kaliteli hem de uygun fiyatlı sütyenler üretmektedir. Sütyene ek olarak sütyen takımı ve jartiyer gibi ürünleri de mevcuttur. Özellikle Avrupa ve Orta Doğu'da çokça tercih edilmektedir.

    yeni inci sütyen kaliteyi ucuz olarak sizlere ulaştırmaktadır. Çok çeşitli sütyen varyantları mevcuttur. iç giyime damga vuran markalardan biridir ve genellikle Avrupa'da ismi sıklıkla duyulur.

    iç giyim ürünlerine her zaman dikkat etmemiz gerekmektedir. Üretimde kullanılan malzemelerin kullanım oranları, kumaşın esnekliği, çekmezlik testi gibi birçok unsuru aynı anda değerlendirerek seçim yapmalıyız.

    iç giyim bayanların erkeklere göre daha dikkatli oldukları bir alandır. Erkeklere göre daha özenli ve daha seçici davranırlar. Biliyorlar ki iç giyimde kullandıkları şeyler kafalarındaki ve ruhlarındaki özellikleri dışa vururlar.

    ReplyDelete
  2. cami avizesi - no deposit bonus forex 2021 - takipçi satın al - takipçi satın al - takipçi satın al - takipcialdim.com/tiktok-takipci-satin-al/ - instagram beğeni satın al - instagram beğeni satın al - btcturk - tiktok izlenme satın al - sms onay - youtube izlenme satın al - no deposit bonus forex 2021 - tiktok jeton hilesi - tiktok beğeni satın al - binance - takipçi satın al - uc satın al - sms onay - sms onay - tiktok takipçi satın al - tiktok beğeni satın al - twitter takipçi satın al - trend topic satın al - youtube abone satın al - instagram beğeni satın al - tiktok beğeni satın al - twitter takipçi satın al - trend topic satın al - youtube abone satın al - takipcialdim.com/instagram-begeni-satin-al/ - tiktok takipçi satın al - tiktok beğeni satın al - twitter takipçi satın al - trend topic satın al - youtube abone satın al - instagram beğeni satın al - perde modelleri - instagram takipçi satın al - takipçi satın al - instagram takipçi satın al

    ReplyDelete
  3. I think you should make video for youtube about it and get some likes from this site https://viplikes.co.za

    ReplyDelete
  4. Ah, there is a good discussion about this paragraph in this place on this website. I read all that, so now I am commenting here too... Qualified travelers to Turkey by filling an online form with their personal details can apply visa to Turkey and pay visa cost Turkey online.

    ReplyDelete
  5. I like your blog! I think you should lead instagram blog. You can buy instagram followers to promote your blog

    ReplyDelete
  6. Thank you for this amazing post.. Wish to travel to Azerbaijan for business and tourism purpose then apply for Azerbaijan visa online. That offers you fast and very secure process.

    ReplyDelete
  7. Hmm , very interesting thoughts! I think In order for more interested people to read your thoughts, you should post it on Instagram, and use the services of https://viplikes.net/ to quickly increase the amount of followers.

    ReplyDelete