How We Validate FunderRank
Rigorous holdout testing on real-world funding data
The Challenge
Most foundation matching tools rely on keyword or semantic search to connect organizations with potential funders. These approaches match based on what foundations and nonprofits say about themselves. SciRise takes a fundamentally different approach: we analyze actual funding relationships to predict where money will flow next.
But how do you know if a prediction model actually works? You test it on real data it has never seen before.
How We Test
We trained our model on historical IRS 990 and 990-PF filing data, then tested whether it could predict new funding relationships that actually appeared in a future year the model never saw. Critically, we only count genuinely new relationships — if a foundation already funded an organization in the training period, a repeat grant doesn't count. The model has to identify funders the organization has never received money from before.
Validation Design
- Test setup: Nonprofits with three or more existing funders; two years of training data, one holdout year; replicated across three test windows
- Training and test data: IRS 990/990-PF filings; the holdout year is never seen during model development
- Population: Nonprofits with three or more existing funders that added a new one. Funders SciRise does not list, such as pure donor-advised fund sponsors, are not counted.
- Success metric: Did at least one of the model's top 10 recommendations appear as a new funder in the test year?
Results
Among nonprofits with three or more existing funders that added a new funder, SciRise placed at least one eventual new funder in its top 10 for 39% of those nonprofits, nearly 2 in 5.
39%
Top-10 hit rate
10x+
Stronger than matching on published mission statements
p < 0.0001
Statistical significance
By comparison, AI mission-matching and keyword models performed far worse on the same test. Our result is more than 10 times the top-10 hit rate of matching on funders' published mission statements, and the difference is statistically significant at p < 0.0001. That comparison uses state-of-the-art AI text embeddings to match each nonprofit's mission to funders' published mission statements, a stronger test than keyword search or category matching.
Why This Matters
A 39% hit rate means the model isn't guessing. When SciRise surfaces a foundation as a top prospect, there's a meaningful, validated probability that foundation will fund an organization like yours. This is the difference between a search tool and a prediction engine.
What Makes This Rigorous
- Strict separation: The model never sees the test year during training, preventing data leakage
- New relationships only: Repeat grants are excluded, so the model must predict genuinely new funding
- Head-to-head comparison: Every nonprofit is scored by both the network model and the search baseline on the same data, enabling direct comparison
- Consistent results: Results replicate across three independent test windows
- Transparent denominator: The headline metric is measured among nonprofits that added at least one new funder in the test year
Data Foundation
Our model is built on 2.6M distinct funder-to-recipient relationships extracted from IRS 990 and 990-PF filings, connecting 19,000+ grantmaking entities to hundreds of thousands of nonprofit recipients. This is not a sample or estimate. Every relationship represents an actual grant reported to the IRS.
We map every private foundation distributing $1M or more a year, giving SciRise the most comprehensive landscape view of foundation giving available.