Character limit on queries?

[azak24]

Hello, I'm having some trouble with getting my particular search query to run. It will work sometimes, and other times will produce the Bad Request (HTTP 400) error. After some isolating of the issue, I'm wondering if it's related to the length of my query.

My query is fairly long, and is made of 4 search terms with AND connecting them.

First, I tested each search term separately, checking for syntax errors.

term1 <- 'TITLE-ABS-KEY("insect communit*" OR "insect abundance" OR "insect diversity" OR pollinator* OR "pollinating insect*" OR apoidea* OR bumblebee* OR honeybee* OR bombus OR bees OR bee OR hymenoptera* OR butterfly OR butterflies OR caterpillar* OR moth OR moths OR lepidoptera* OR fly OR flies OR diptera* OR beetle* OR coleoptera* OR ant OR ants OR formicidae*)'
term2 <- 'TITLE-ABS-KEY("anthropogenic disturbance*" OR "habitat loss" OR "natural resource extraction" OR "resource extraction" OR "energy development" OR "natural gas" OR "fossil fuel*" OR "well pad*" OR "crude oil" OR "oil and gas" OR "oil and natural gas" OR pipeline* OR drilling OR "oil well*" OR road OR roads OR roadside OR traffic OR OHV OR "off-highway vehicle" OR off-road OR trampl* OR "air pollut*" OR "nitrogen oxide*" OR exhaust OR diesel OR ozone OR nitrate* OR "heavy metal*" OR "air turbulence" OR (road* AND dust) OR "dust pollution" OR "dust interference" OR "dust reduction" OR "dust control" OR particulate* OR "artificial light*" OR "light pollution" OR "noise pollution" OR "anthropogenic noise" OR vibration* OR "sensory pollution" OR "ecological restoration" OR reclamation OR reclaimed OR "noxious weed*" OR "invasive weed*" OR "non-native plant*" OR "non-native grass*" OR "non-native forb*" OR "invasive species" OR "invasive plant*" OR "invasive grass*" OR "invasive forb*")'
term3 <- 'TITLE-ABS-KEY(mitigat* OR disturbance* OR disturbed OR degradation OR degrade* OR (soil AND compaction) OR (soil AND erosion) OR (soil AND stabil*) OR remediation OR revegetation OR seeded OR seeding OR "seed mix*" OR respread* OR unmound* OR (road* AND anti-icing) OR (road* AND de-icing) OR (road* AND adjuvent*) OR (road* AND salt*) OR "magnesium chloride" OR bioaccumulat*)'
term4 <- 'TITLE-ABS-KEY("ecosystem service*" OR "flowering plant*" OR "native plant*" OR "native vegetation" OR "native forb*" OR "annual forb*" OR "perennial forb*" OR "native grass*" OR "perennial grass*" OR "fruit set" OR "seed set" OR nectar OR "floral resource*" OR pollen OR photosynthesi* OR mortality OR "acute toxicity" OR "direct mortality" OR abundance OR "shannon diversity" OR "functional diversity" OR "community richness" OR "species richness" OR "species diversity" OR biodiversity OR "pollinator behavior*" OR "pollinator activity" OR pollination OR "plant–pollinator interaction*" OR "buzz pollination" OR sonication OR "pollinator visit*" OR "visitation rate" OR "foraging behavior*" OR navigat* OR *nesting OR "physiological* stress*" OR "olfactory cue*" OR "floral scent*" OR "floral VOC*" OR "volatile organic compound*" OR "chemical attract*" OR "visual cue*" OR ultraviolet OR UV OR habitat)'
searchOut <- scopus_search(query = term1, view = "STANDARD", count = 1, max_count = 10, start = 0)
searchOut <- scopus_search(query = term2, view = "STANDARD", count = 1, max_count = 10, start = 0)
searchOut <- scopus_search(query = term3, view = "STANDARD", count = 1, max_count = 10, start = 0)
searchOut <- scopus_search(query = term4, view = "STANDARD", count = 1, max_count = 10, start = 0)

All four processed successfully.

Then I combined the 4 sections into the full search term, using either the paste0 function or just typing the whole string (no difference in result).

fullQuery <- paste0(term1," AND ",term2," AND ",term3," AND ",term4)
#fullQuery <- 'TITLE-ABS-KEY("insect communit*" OR "insect abundance" OR "insect diversity" OR pollinator* OR "pollinating insect*" OR apoidea* OR bumblebee* OR honeybee* OR bombus OR bees OR bee OR hymenoptera* OR butterfly OR butterflies OR caterpillar* OR moth OR moths OR lepidoptera* OR fly OR flies OR diptera* OR beetle* OR coleoptera* OR ant OR ants OR formicidae*) AND TITLE-ABS-KEY("anthropogenic disturbance*" OR "habitat loss" OR "natural resource extraction" OR "resource extraction" OR "energy development" OR "natural gas" OR "fossil fuel*" OR "well pad*" OR "crude oil" OR "oil and gas" OR "oil and natural gas" OR pipeline* OR drilling OR "oil well*" OR road OR roads OR roadside OR traffic OR OHV OR "off-highway vehicle" OR off-road OR trampl* OR "air pollut*" OR "nitrogen oxide*" OR exhaust OR diesel OR ozone OR nitrate* OR "heavy metal*" OR "air turbulence" OR (road* AND dust) OR "dust pollution" OR "dust interference" OR "dust reduction" OR "dust control" OR particulate* OR "artificial light*" OR "light pollution" OR "noise pollution" OR "anthropogenic noise" OR vibration* OR "sensory pollution" OR "ecological restoration" OR reclamation OR reclaimed OR "noxious weed*" OR "invasive weed*" OR "non-native plant*" OR "non-native grass*" OR "non-native forb*" OR "invasive species" OR "invasive plant*" OR "invasive grass*" OR "invasive forb*") AND TITLE-ABS-KEY(mitigat* OR disturbance* OR disturbed OR degradation OR degrade* OR (soil AND compaction) OR (soil AND erosion) OR (soil AND stabil*) OR remediation OR revegetation OR seeded OR seeding OR "seed mix*" OR respread* OR unmound* OR (road* AND anti-icing) OR (road* AND de-icing) OR (road* AND adjuvent*) OR (road* AND salt*) OR "magnesium chloride" OR bioaccumulat*) AND TITLE-ABS-KEY("ecosystem service*" OR "flowering plant*" OR "native plant*" OR "native vegetation" OR "native forb*" OR "annual forb*" OR "perennial forb*" OR "native grass*" OR "perennial grass*" OR "fruit set" OR "seed set" OR nectar OR "floral resource*" OR pollen OR photosynthesi* OR mortality OR "acute toxicity" OR "direct mortality" OR abundance OR "shannon diversity" OR "functional diversity" OR "community richness" OR "species richness" OR "species diversity" OR biodiversity OR "pollinator behavior*" OR "pollinator activity" OR pollination OR "plant–pollinator interaction*" OR "buzz pollination" OR sonication OR "pollinator visit*" OR "visitation rate" OR "foraging behavior*" OR navigat* OR *nesting OR "physiological* stress*" OR "olfactory cue*" OR "floral scent*" OR "floral VOC*" OR "volatile organic compound*" OR "chemical attract*" OR "visual cue*" OR ultraviolet OR UV OR habitat)'
nchar(fullQuery)
searchOut <- scopus_search(query = fullQuery, view = "STANDARD", count = 1, max_count = 10, start = 0)

The result was "Error in get_results(query, start = init_start, count = count, verbose = verbose, : Bad Request (HTTP 400)."

So I started chipping away at the full query, removing terms from the end.

shortenedQuery <- 'TITLE-ABS-KEY("insect communit*" OR "insect abundance" OR "insect diversity" OR pollinator* OR "pollinating insect*" OR apoidea* OR bumblebee* OR honeybee* OR bombus OR bees OR bee OR hymenoptera* OR butterfly OR butterflies OR caterpillar* OR moth OR moths OR lepidoptera* OR fly OR flies OR diptera* OR beetle* OR coleoptera* OR ant OR ants OR formicidae*) AND TITLE-ABS-KEY("anthropogenic disturbance*" OR "habitat loss" OR "natural resource extraction" OR "resource extraction" OR "energy development" OR "natural gas" OR "fossil fuel*" OR "well pad*" OR "crude oil" OR "oil and gas" OR "oil and natural gas" OR pipeline* OR drilling OR "oil well*" OR road OR roads OR roadside OR traffic OR OHV OR "off-highway vehicle" OR off-road OR trampl* OR "air pollut*" OR "nitrogen oxide*" OR exhaust OR diesel OR ozone OR nitrate* OR "heavy metal*" OR "air turbulence" OR (road* AND dust) OR "dust pollution" OR "dust interference" OR "dust reduction" OR "dust control" OR particulate* OR "artificial light*" OR "light pollution" OR "noise pollution" OR "anthropogenic noise" OR vibration* OR "sensory pollution" OR "ecological restoration" OR reclamation OR reclaimed OR "noxious weed*" OR "invasive weed*" OR "non-native plant*" OR "non-native grass*" OR "non-native forb*" OR "invasive species" OR "invasive plant*" OR "invasive grass*" OR "invasive forb*") AND TITLE-ABS-KEY(mitigat* OR disturbance* OR disturbed OR degradation OR degrade* OR (soil AND compaction) OR (soil AND erosion) OR (soil AND stabil*) OR remediation OR revegetation OR seeded OR seeding OR "seed mix*" OR respread* OR unmound* OR (road* AND anti-icing) OR (road* AND de-icing) OR (road* AND adjuvent*) OR (road* AND salt*) OR "magnesium chloride" OR bioaccumulat*) AND TITLE-ABS-KEY("ecosystem service*" OR "flowering plant*" OR "native plant*" OR "native vegetation" OR "native forb*" OR "annual forb*" OR "perennial forb*" OR "native grass*" OR "perennial grass*" OR "fruit set" OR "seed set" OR nectar OR "floral resource*" OR pollen OR photosynthesi* OR mortality OR "acute toxicity" OR "direct mortality" OR abundance OR "shannon diversity" OR "functional diversity" OR "community richness" OR "species richness" OR "species diversity" OR biodiversity)'
nchar(shortenedQuery)
searchOut <- scopus_search(query = shortenedQuery, view = "STANDARD", count = 1, max_count = 10, start = 0)

A 2232-character term did not run, but removing the last term, "biodiversity", to get to 2216 characters, DOES run successfully.

shortenedQuery <- 'TITLE-ABS-KEY("insect communit*" OR "insect abundance" OR "insect diversity" OR pollinator* OR "pollinating insect*" OR apoidea* OR bumblebee* OR honeybee* OR bombus OR bees OR bee OR hymenoptera* OR butterfly OR butterflies OR caterpillar* OR moth OR moths OR lepidoptera* OR fly OR flies OR diptera* OR beetle* OR coleoptera* OR ant OR ants OR formicidae*) AND TITLE-ABS-KEY("anthropogenic disturbance*" OR "habitat loss" OR "natural resource extraction" OR "resource extraction" OR "energy development" OR "natural gas" OR "fossil fuel*" OR "well pad*" OR "crude oil" OR "oil and gas" OR "oil and natural gas" OR pipeline* OR drilling OR "oil well*" OR road OR roads OR roadside OR traffic OR OHV OR "off-highway vehicle" OR off-road OR trampl* OR "air pollut*" OR "nitrogen oxide*" OR exhaust OR diesel OR ozone OR nitrate* OR "heavy metal*" OR "air turbulence" OR (road* AND dust) OR "dust pollution" OR "dust interference" OR "dust reduction" OR "dust control" OR particulate* OR "artificial light*" OR "light pollution" OR "noise pollution" OR "anthropogenic noise" OR vibration* OR "sensory pollution" OR "ecological restoration" OR reclamation OR reclaimed OR "noxious weed*" OR "invasive weed*" OR "non-native plant*" OR "non-native grass*" OR "non-native forb*" OR "invasive species" OR "invasive plant*" OR "invasive grass*" OR "invasive forb*") AND TITLE-ABS-KEY(mitigat* OR disturbance* OR disturbed OR degradation OR degrade* OR (soil AND compaction) OR (soil AND erosion) OR (soil AND stabil*) OR remediation OR revegetation OR seeded OR seeding OR "seed mix*" OR respread* OR unmound* OR (road* AND anti-icing) OR (road* AND de-icing) OR (road* AND adjuvent*) OR (road* AND salt*) OR "magnesium chloride" OR bioaccumulat*) AND TITLE-ABS-KEY("ecosystem service*" OR "flowering plant*" OR "native plant*" OR "native vegetation" OR "native forb*" OR "annual forb*" OR "perennial forb*" OR "native grass*" OR "perennial grass*" OR "fruit set" OR "seed set" OR nectar OR "floral resource*" OR pollen OR photosynthesi* OR mortality OR "acute toxicity" OR "direct mortality" OR abundance OR "shannon diversity" OR "functional diversity" OR "community richness" OR "species richness" OR "species diversity")'
nchar(shortenedQuery)
searchOut <- scopus_search(query = shortenedQuery, view = "STANDARD", count = 1, max_count = 10, start = 0)

If there is a character limit, it doesn't seem to be a hard one; I tested a query with 2200 characters that didn't work, for example.

Lastly, I'll note that I can run the full query in Advanced Search on the Scopus database webpage and get 2,748 hits.

Is there something else going on here, or do you think the error is related to the length of the query?