Streamlining AI Paper Discovery: Building an Automated Research Newsletter
Leveraging Claude to discover and absorb more relevant research
I’m going to take a quick interlude from my ongoing blog series to share a fun side project I just completed - building an AI research paper newsletter!
With the increasing volume of AI research being published, I found myself wanting a more automated way to discover papers aligned with my interests in practical AI implementation and regulation, among other LLM-related things. While there are excellent existing tools and curated newsletters, I wanted something tailored specifically to my research priorities – and I wanted to experiment with using Claude for content analysis.
Try It Out!
If you're interested in seeing how this works in practice:
Check out the Github repository
A Simple Solution for Paper Discovery
The system is fully automated, and executes in the following manner:
1. Monitors arXiv RSS feeds for new AI papers
2. Uses Claude to evaluate their relevance to my interests
3. Generates summaries focused on key findings and practical implications
4. Delivers curated papers via email twice weekly
The system runs entirely on AWS and is relatively lightweight and very inexpensive, using:
Lambda functions for paper processing and the subscription workflows
Claude for content analysis
SES for email delivery
DynamoDB for subscriber management and S3 for email storage
Cloudwatch and SNS for monitoring
(Screenshot generated by Claude in the browser - not my favorite AWS diagram, but Claude tried its best).
How It Works
The core functionality revolves around Claude analyzing papers through two lenses:
Initial Screening
Claude first evaluates titles and abstracts to identify papers that match specified interests. In my case, this includes papers on:
Data-driven evaluation frameworks
Production deployment challenges
Real-world implementation architectures
Regulatory approaches
Economic impact analysis
Domain-specific LLM applications, like in healthcare
I also provide some example papers that I have been interested in lately as a means to provide more context to the general criteria provided to Claude.
Detailed Analysis
For papers that pass the initial screening, Claude generates:
A concise summary of key findings
Practical implications
A link to the full paper for a deeper dive
I’m using the HTML link embedded in each arXiv paper page to parse the full text, so if that is not provided for a specific paper, I include the link as an additional paper to review.
The Newsletter Format
Each issue includes:
Featured papers with detailed summaries and links to the full paper
Additional papers of interest when the HTML is not available
An overview of emerging trends across the selected papers
I’m now absorbing more research with this newsletter than I was previously, and notably more aware of a broader set of developments. I’m still clicking into papers that I’m really interested in to go beyond the summary provided, so it’s been a great way to get “breadth” while retaining “depth.”
Open Source and Adaptable
I've made the code available on GitHub for anyone interested in creating their own version. You can take or leave the newsletter portion, but at minimum can turn it a personalized way to sort through papers using Claude.
The repository includes:
Complete AWS infrastructure as code (CloudFormation)
Paper analysis and newsletter generation logic - customize this to your interests
Email templates and subscription management
Detailed deployment instructions
There’s likely some manual updates required to get it working in your environment, but this should get most of the way there.
Looking Forward
This project serves as a practical example of using LLMs for content analysis and curation. While it's a relatively simple implementation, it demonstrates how tools like Claude can be leveraged to streamline information discovery and processing.
I'm continuing to refine the system based on feedback and my own evolving research interests. If you have suggestions for improvements or interesting use cases, feel free to contribute to the repository or reach out with ideas! Subscribe or contact me at my personal webpage.