From text to token: How tokenization pipelines work
Understanding how search engines transform text into tokens through character filtering, tokenization, stemming, and stopword removal.
The latest blog posts to take off on Hacker News. 7,293 posts.
Understanding how search engines transform text into tokens through character filtering, tokenization, stemming, and stopword removal.
Today, we’re pleased to release pg_clickhouse, an Apache 2-licensed PostgreSQL extension to transparently execute analytics queries on ClickHouse directly from PostgreSQL.
One of legal tech's clichés is that "lawyers live in Word". This is demonstrably incorrect. I, for example, am a lawyer and in fact live in London, England…
Recent advancements in Cloudflare Python Workers means fast cold starts, comprehensive package support, and a great developer experience. We explain how they were achieved and show how Python can be used to build serverless applications on Cloudflare.
After exploring Java bytecode in previous years (2022, 2023, 2024), this year we’ll take an unexpected detour for a Java advent: instead of generating Java bytecode, we’ll use Java to build and execute LLVM IR, the intermediate language behind compilers like clang. Using Java’s Foreign Function & Memory (FFM) API…
An exploration of Go's portability limits when integrating with system-specific C libraries like libsystemd and dealing with glibc vs. musl systems.
Large language models (LLMs) are an indisputable breakthrough of the last five years, potentially profoundly changing the way that we work. As with any extraordinarily powerful tool, LLM use has both promise and peril — and that they are so general-purpose leaves real questions about how and when they should be used…
We chose Zig over C++ and Rust because we wanted a simple, modern systems language. Here's what we learned building a browser with it.
I have a hobby. Whenever I see the comment // this should never happen in code, I try to find out the exact conditions under which it could happen. And in 90% of cases, I find a way to do just that. More often than not, the developer just hasn’t cons…
Bun has been acquired by Anthropic. Anthropic is betting on Bun as the infrastructure powering Claude Code, Claude Agent SDK, and future AI coding products & tools.
Earlier this year, the TypeScript team announced that we've been porting the compiler and language service to native code to take advantage of better raw
The most powerful AI platform for enterprises. Customize, fine-tune, and deploy AI assistants, autonomous agents, and multimodal AI with open models.
zigtools: We make free and open-source tools for the @ziglang community.
Electron is the most popular framework for building desktop apps using web technologies. In this article, we'll discuss a newer alternative, Tauri, and weigh some of the pros and cons between the two frameworks.
How an initializer agent and an incremental coding agent help the Claude Agent SDK make steady progress across many context windows.
Consider this tale of I/O and performance. We’ll start with blocking I/O, explore io_uring and kqueue, and take home an event loop very similar to some software you may find familiar.
Initial kernel and subsystem support for new Snapdragon 8 Elite Gen 5 posted for review. Learn what’s in the patches and how you can start working with them.
In the book Flatland: A Romance of Many Dimensions, a two-dimensional world called “Flatland” is inhabited by polygonal creatures like triangles, squares, and circles. The protaganist, a square, is…
Malware driving attack includes "dead man's switch" that can harm user data.
The Zig community is decentralized. Anyone is free to start and maintain their own community , which is not subject to these rules.
See how PostgreSQL 19 speeds up aggregations by aggregating before joins, with a benchmark showing over 5x faster execution.
An exploration of how .NET evolved from a distributed build system to Unified Build, dramatically reducing complexity and build times while improving flexibility and predictability for shipping .NET releases.
After years of engineering, design, and community collaboration, we're excited to release Unison 1.0. This version delivers a refined programming workflow and a mature toolchain. Join us as we celebrate this milestone and look ahead to the future of Unison.
We're excited to announce a major architectural improvement to Atuin Desktop: a completely redesigned runbook execution engine.
LE Audio replaces classic Bluetooth® audio profiles with a modular, low-energy architecture.
Claude can now discover, learn, and execute tools dynamically to enable agents that take action in the real world. Here’s how.
Over 1,000 NPM packages were infected using the same method as the previous attack, infecting with a fake Bun runtime. The attacker leveraged the `preinstall` script to download and execute a malicious payload, stealing sensitive information from the compromised machines.