How to Crawl the Web with Scrapy
How to use the python web scraping framework Scrapy to crawl indeed.com. Learn data engineering strategies for getting actionable insights from public information.
Every blog post that took off on Hacker News, ranked by the votes of its best submission. 49,247 posts.
How to use the python web scraping framework Scrapy to crawl indeed.com. Learn data engineering strategies for getting actionable insights from public information.
There’s a concept from computer security known as a class break. It’s a particular security vulnerability that breaks not just one system, but an entire class of systems. Examples might be a vulnerability in a particular operating system that allows an attacker to take remote control of every computer that runs on…
Ever wanted to know how exactly did Quake’s precomputed visibility work? I did, so I wrote vis.py , a reimplementation of their algorithm in Python. This guide has all the information you need to understand vis , the tool used by Quake, Half-Life and Source Engine games.
Some time ago, I came across the Wikipedia article of Niels Henrik Abel , and something there burnt its way into my mind. Abel changed the face of mathematics, despite dying at 27. And here comes the thing – I give the mic to Wikipedia now:
Wayland is all the rage those days. Distributions left and right switch to it, many readers of my previous article on writing a X11 GUI from scratch in x86_64 assembly asked for a follow-up article about Wayland, and I now run Wayland on my desktop. So here we go, let's write a (very simple) GUI program with Wayland…
Although the presence of ET planetary-based life as we know life and don’t know it, is all, comparatively as, and less and in gradations more technologically and cognitively achieved or advanced than our species, such life, in mathematical and thermochemical terms, is a must on a vast in raw numbers but quite possibly…
You can make your own website. This guide can help anyone make a website with HTML and CSS. You can make a personal site, link tree, hobby site, digital garden, fansite, or any other website you want. It's fun, and easy enough for a 12-year-old.
I recently had solar panels and a battery storage system from GivEnergy installed at my house. A major selling point for me was that they have a local network API which can be used to monitor and…
You're probably reading this page because you've attempted to access some part of my blog (Wandering Thoughts) or CSpace , the wiki thing it's part of. Unfortunately you're using a browser version that my anti-crawler precautions consider suspicious, most often because it's too old (most often this applies to versions…
Exploratory interfaces for navigating photos
One day in January 2013 I found myself wasting time on the internet.
A guide to AI and LLM evals for engineers and product managers. Learn how to test AI systems, analyze failures, and improve retrieval, agents, and AI products.
As I continue reading An Introduction to Parallel Programming, I cannot help but notice a connection between communication among processors, communication among human beings, and communication within the individual self....
As a resident member of the EEVblog Test Equipment Anonymous support group, I have picked up my fair share of old test equipment boat anchors, but my knees buckle at broken recently manufactured oscilloscopes in the sub $200 range. In this case, the Agilent / Keysight Infiniivision X-series oscilloscopes have been an…
I recently wanted to take a hierarchy of items and draw them in a nice tree structure. For example, a Family Tree. At the time, I thought “This will be easy, I’ll just Google for an alg…
Blog posts and articles by author and game developer Jordan Mechner
my favorite form of entertainment is downloading PDFs ✦ plus favorite Fluxus artists and early programs
Clarifying the shebang (#!) mechanism: A step-by-step look using strace and kernel code shows how Linux handles script execution directly, revealing the shell isn't involved initially.
Manipulating actually-big-data is just as easy as performing an analysis on a dataset with only a few records.
You’re in the iPhone mail app, and you’re looking at your inbox, when a new email arrives. What happens?
One thing you may not know about me is that I am a writer. More specifically, I'm a freelance writer. This means I can have several clients at once, work from home in my underwear, and set my own hours. It is basically like being a prostitute, but with less stringent hygiene requirements.