Depending on the type of API used by your data provider, there may be caps to the number of rows you can pull by default.
Here is how to change the limit when pulling data from Socrata portals, which is one the most common open data API provider in the U.S.
Limit
The $limit parameter controls the total number of rows returned, and it defaults to 1,000 records per request. For example, if you wanted to only return the top ten strongest earthquakes, you could use $limit , simply by appending it to the URL:
https://soda.demo.socrata.com/resource/4tka-6guv.json?$limit=10 But which rows are you getting? The most recent, the oldest? The 10 largest magnitudes? In order to control what rows you are getting, you need to use $order .
Order
The $order parameter determines how the rows should be sorted before you pull them, using the values from the specified columns. Sorting can be performed in either ascending or descending order, the default being ascending, but you can also reverse the order with DESC.
For example, to sort our earthquakes by magnitude, in descending order:
https://soda.demo.socrata.com/resource/4tka-6guv.json?$order=magnitude%20DESCNow let's use $limit in conjunction with $order:
https://soda.demo.socrata.com/resource/4tka-6guv.json?$order=magnitude%20DESC&$limit=10Note: Depending on the version of the Socrata API endpoint, it will have different maximums for $limit:
- Version 2.0 endpoints have a maximum $limit of 50,000
- Version 2.1 endpoints have no maximum
Details are available in the API documentation for each API. Make sure you pick a limit appropriate to the speed of your connection, as HTTP calls will time out and payloads for high $limits can be very large.