Rigorous holdout testing on real-world funding data
The Challenge
Most foundation matching tools rely on keyword or semantic search to connect organizations with potential funders. These approaches match based on what foundations and nonprofits say about themselves. SciRise takes a fundamentally different approach: we analyze actual funding relationships to predict where money will flow next.
But how do you know if a prediction model actually works? You test it on real data it has never seen before.
How We Test
We trained our model on historical IRS 990 and 990-PF filing data, then tested whether it could predict new funding relationships that actually appeared in a future year the model never saw. Critically, we only count genuinely new relationships — if a foundation already funded an organization in the training period, a repeat grant doesn't count. The model has to identify funders the organization has never received money from before.
Validation Design
- Training data: Multiple years of IRS 990/990-PF filings covering 9.8M foundation grants
- Test data: A later holdout year, never seen during model development
- Success metric: Did at least one of the model's top 10 recommendations appear as a new funder in the test year?
- Robustness: Replicated across multiple test windows with consistent results
Results
Among nonprofits that added a new funder, SciRise placed at least one eventual new funder in its top 10 for 31% of those nonprofits, nearly 1 in 3.
31%
Top-10 hit rate
10x+
Stronger than mission-alignment matching
p < 0.0001
Statistical significance
By comparison, the best AI mission-matching and keyword models performed far worse on the same test. Our result is more than 10x the top-10 hit rate of the strongest tested mission-alignment baseline, and the difference is statistically significant at p < 0.0001. The baseline uses state-of-the-art AI text embeddings to match foundation and nonprofit profiles, a stronger test than keyword search or category matching.
Why This Matters
A 31% hit rate means the model isn't guessing. When SciRise surfaces a foundation as a top prospect, there's a meaningful, validated probability that foundation will fund an organization like yours. This is the difference between a search tool and a prediction engine.
What Makes This Rigorous
- Strict separation: The model never sees the test year during training, preventing data leakage
- New relationships only: Repeat grants are excluded, so the model must predict genuinely new funding
- Head-to-head comparison: Every nonprofit is scored by both the network model and the search baseline on the same data, enabling direct comparison
- Consistent results: Results replicate across multiple independent test periods
- Transparent denominator: The headline metric is measured among nonprofits that added at least one new funder in the test year
Data Foundation
Our model is built on 2.3M distinct funder-to-recipient relationships extracted from IRS 990 and 990-PF filings, connecting 19,000+ grantmaking foundations to hundreds of thousands of nonprofit recipients. This is not a sample or estimate. Every relationship represents an actual grant reported to the IRS.
We map every private foundation distributing $1M or more annually, giving SciRise the most comprehensive view of foundation giving available.