THE APISS QUESTIONNAIRE: A NEW TOOL TO ASSESS THE EPIDEMIOLOGY  
OF SYSTEMIC ALLERGIC REACTIONS TO BEE VENOM IN BEEKEEPERS
Tanja CARLI
 1,2*
   , Mitja KOŠNIK
 3,4
   , Lijana ZALETEL KRAGELJ
 1,5
   ,  
Genc BURAZERI
 6,7
   , Andreja KUKEC
 1,5
 
1 
National Institute of Public Health, Trubarjeva cesta 2, 1000 Ljubljana, Slovenia
2 
University of Ljubljana, Faculty of Medicine, Vrazov trg 2, 1000 Ljubljana, Slovenia
3 
University Clinic of Respiratory and Allergic Diseases Golnik, Golnik 36, 4204 Golnik, Slovenia
4 
University of Ljubljana, Faculty of Medicine, Chair of Internal Medicine, Zaloška cesta 7, 1000 Ljubljana, Slovenia
5 
University of Ljubljana, Chair of Public Health, Zaloška cesta 4, 1000 Ljubljana, Slovenia
6 
Department of International Health, School CAPHRI (Care and Public Health 
Research Institute), Maastricht University, Maastricht, the Netherlands
7 
Department of Public Health, Faculty of Medicine, University of Medicine, Tirana, Albania
Received: Nov 10, 2022
Accepted: May 5, 2023
Original scientific article
*Correspondence: tanja.carli@nijz.si
10.2478/sjph-2023-0019 Zdr Varst. 2023;62(3):137-144
137
VPRAŠALNIK APISS: NOVO ORODJE ZA EPIDEMIOLOŠKO OCENO 
SISTEMSKIH ALERGIJSKIH REAKCIJ PO PIKU ČEBELE MED ČEBELARJI
© National Institute of Public Health, Slovenia. 
Carli T, Košnik M, Zaletel-Kragelj L, Burazeri G, Kukec A. The APISS questionnaire: A new tool to assess the epidemiology of systemic allergic reactions to bee venom in beekeepers. 
Zdr Varst. 2023;62(3):137-144. doi: 10.2478/sjph-2023-0019.
ABSTRACT
Keywords: 
Hymenoptera
Hypersensitivity
Beekeeping
Questionnaire
IZVLEČEK
Ključne besede: 
kožekrilci
preobčutljivost
čebelarjenje
vprašalniki
Aim: To develop and validate a comprehensive questionnaire to be used as an instrument in cross-sectional 
studies among beekeepers. 
Methods: A comprehensive questionnaire in Slovenian was validated by an expert panel (n=13) for content 
relevance and a rater panel (n=14) for clarity and comprehensibility. The content validity indices (an item-level 
content validity index and scale-level content validity index based on the average and universal agreement 
method) and item-level face validity index were calculated in accordance with the recommended number of 
both the review panels with their implications on the acceptable cut-off scores. Piloting was performed in a 
sample (n=50) of the target population (N=1.080) using telephone interviews. 
Results: The item-level content validity index and scale-level content validity index based on the average 
method exhibited excellent content validity (0.97), while the scale-level content validity index based on the 
universal agreement method reached a value of 0.72. The item-level face validity index of 1.00 indicated that 
all items were clear and comprehensive. 
Conclusions: The new instrument may be considered valid and feasible for use in nationwide population-based 
studies among Slovenian beekeepers and eventually in other populations.
Namen: Razviti celostni vprašalnik in preveriti njegovo veljavnost na način, da bo uporaben kot orodje v 
presečnih raziskavah med čebelarji.
Metode: Celostni vprašalnik v slovenskem jeziku smo oblikovali na podlagi sistematičnega pregleda literature. 
Protokol za sistematični pregled literature smo predhodno registrirali v mednarodni prospektivni register za 
sistematične preglede in pri pripravi upoštevali Prednostna poročila za sistematične preglede in metaanalizo 
(2022). Vsebinsko veljavnost vprašalnika smo ugotavljali na podlagi strinjanja skupine strokovnjakov različnih 
strok ( n = 1 3 ) , zdravorazumsko vel javnost pa znotra j skupine čebelar jev laikov  ( n = 1 4 ) . Indekse vsebinske 
veljavnosti (indeks vsebinske veljavnosti na ravni postavke, indeks vsebinske veljavnosti na ravni lestvice, 
izračunan z metodo povprečnega in splošnega soglasja) in indeks zdravorazumske veljavnosti na ravni postavke 
smo izračunali glede na priporočeno število oseb obeh skupin, ki sta bili vključeni v preverjanje veljavnosti 
vprašalnika. Pilotno testiranje vprašalnika smo izvedli na vzorcu (n = 50) ciljne populacije čebelarjev (N = 1.080) 
z metodo telefonskega intervjuja.
Rezultati: Indeks vsebinske veljavnosti na ravni postavk in indeks vsebinske veljavnosti na ravni lestvice, 
izračunana z metodo povprečnega soglasja, sta znašala 0,97; indeks vsebinske veljavnosti na ravni lestvice, 
izračunan z metodo splošnega soglasja, pa 0,72. Vrednost indeksa zdravorazumske veljavnosti na ravni postavke 
1,00 nakazuje, da so vse postavke jasne in razumljive.
Zaključki: Rezultati potrjujejo veljavnost in primernost razvitega orodja, ki bi ga lahko uporabili v nacionalnih 
populacijskih raziskavah med slovenskimi čebelarji. Prevod vprašalnika v angleški jezik nakazuje možnost 
njegove širše uporabe na mednarodni ravni.

1 INTRODUCTION
Allergic reaction to Hymenoptera (bees, wasps, and 
hornets) is an important public health problem, with a large 
local reaction (LLR) and systemic allergic reaction (SAR) as 
the two main clinical presentations. LLR is defined as local 
swelling exceeding 10 cm in diameter, increasing within 
24–48 hours after the sting, and lasting longer than 72 
hours, while SAR involves one or more organs and consists 
of various grades, depending of the classification used (1). 
Stinging Hymenoptera are amongst the most common 
causes of SAR (2), with bees of the family Apidae and 
wasps of the family Vespinae as the main perpetrators 
of a Hymenoptera venom allergy (HVA) in developed 
countries (3). The prevalence of the self-reported SAR 
to Hymenoptera venom in the adult general population 
across different European countries over a 20-year period 
ranges from 0.9 to 8.9% (4). The rates are the highest 
among beekeepers due to their unavoidable seasonal or 
all-year-round exposure to the elicitors of a HVA, bees 
in particular. However, epidemiological observational 
studies worldwide published between 1996 to 2018 have 
shown significant differences in the prevalence of the self-
reported SAR, with estimates ranging from 2.1 (5) to 37.6% 
(6). The factors accounting for the degree of variability 
have been attributed to the degree of exposure, data 
collection technique used, definition of allergic reaction, 
climate and geographical location (7).
To the best of our knowledge, to date no comprehensive 
and validated questionnaires exist for assessing 
epidemiological data on the self-reported SAR to bee 
venom and the risk factors for SAR among beekeepers. 
Therefore, this study aims to address this gap through 
the development and validation of a comprehensive 
questionnaire to be used as an instrument in cross-
sectional studies among Slovenian beekeepers. The first 
objective of the study is to describe the questionnaire’s 
development process and assess its validity. The second 
objective is to pilot-test the final version of the tool to 
verify its feasibility for the proposed research approach. 
2 METHODS
2.1 Questionnaire development 
2.1.1 Domain identification and item generation 
Initially, a systematic literature review (data not published 
yet) was carried out using the methodology previously 
reported in a protocol paper (8), aiming to identify the 
observed outcome (questionnaires) of the relevant surveys 
to generate a comprehensive new instrument, making it 
possible, through the targeted environmental history, to 
address epidemiological public health aspects of the self-
reported SAR to bee venom and the risk factors for SAR 
10.2478/sjph-2023-0019 Zdr Varst. 2023;62(3):137-144
138
among beekeepers to provide a more complete information 
in the process of assessing clinical aspects and diagnosis 
of HVA. Providing evidence-based data, the following 
objectives were: a) assessing the prevalence of the self-
reported SAR to Hymenoptera venom in beekeepers 
worldwide, b) exploring the impact of environmental 
differences worldwide on the prevalence rates, and c) 
updating the principal risk factors for SAR. In total, of 442 
studies identified, seven original articles were included 
and used for developing a comprehensive questionnaire 
in the Slovenian language. Preliminary data show the 
prevalence of the self–reported SAR to bee venom ranging 
from 2.1 to 37.6%. Several risk factors were identified 
that either increase, decrease risk or no association was 
proven. The identification of pertinent content domains 
and item generation were initially carried out by the first 
author based on two main sources: descriptions of studies’ 
questionnaires and a copy of a German questionnaire 
(Questionnaire for the Assessment of Beekeepers’ Health, 
QABH) obtained on request (9). The pool of domains and 
items were reviewed and refined by the consensus group, 
comprising the three authors of this paper, until complete 
agreement on the domains/items was achieved, resulting 
in a first draft of the study questionnaire. An example 
of the refinement is the question about current smoking 
status (smoking status domain). The predefined responses 
were refined according to the World Health Organization 
Countrywide Integrated Noncommunicable Diseases 
Intervention (CINDI) programme, more specifically 
according to the CINDI Health Monitor Questionnaire (10).
2.2 Validity assessment
2.2.1 Expert panel selection and recruitment 
A panel of 15 experts from different fields (public health, 
sanitary engineering, clinical medicine, veterinary 
medicine, biology, microbiology and beekeeping), was 
nominated and selected based on the criteria presented 
by Fernandez-Gomez (11). All of these individuals were 
knowledgeable about the subject, either because of 
their professional knowledge/academic background or 
because of their epidemiological and/or clinical work 
experience. The recruitment period lasted from 5th to 
10th September 2021. Each nominee was individually and 
confidentially contacted through email and invited to be 
a part of the expert panel. The experts who indicated 
their willingness to participate were emailed again and 
provided with all the documents needed to conduct the 
validity process (i.e., an informed consent form attached 
to an explanatory cover letter, a first version of the 
questionnaire, a brief demographic questionnaire (i.e., on 
gender, age, statistical region, place of residence), and 
a content validation form). The cover letter included a 
short description of the study’s purpose, a description 
of the content review instrument with instructions, a 

10.2478/sjph-2023-0019 Zdr Varst. 2023;62(3):137-144
139
rating score to help guide the experts with regard to the 
scoring method and the reasons for their (the experts’) 
nomination (12).
2.2.2 Content validity
For qualitative validity, members of the expert panel 
were asked to critically review each item of the targeted 
domain prior to scoring its relevance. Following the 
experts’ assessment, domains and items were revised 
by the consensus group and, if necessary, modified upon 
their agreement. 
For the quantitative assessment of the content validity, a 
panel of experts was asked to rate the instrument items 
in terms of their relevance. A four-point Likert scale 
based on a criterion propounded by Davis (13) was used 
as follows: 1 = not relevant to the measured domain; 2 
= somewhat relevant to the measured domain; 3 = quite 
relevant to the measured domain; and 4 = highly relevant 
to the measured domain. The scale was recoded into 
dichotomised relevant (rating of 3 and 4) and non-relevant 
(rating of 1 and 2) values prior to calculating the content 
validity indices (CVIs) (14).
Quantitative data were analysed by the computing CVIs 
at the item (item-level content validity index, I-CVI) and 
scale levels (scale-level content validity index, S-CVI), 
the latter being calculated using two methods, i.e., the 
average (Ave, S-CVI/Ave) and the universal agreement 
(UA, S-CVI/UA) (15). Calculations were made using the 
following equations (1-3): 
2.2.3 Pre-piloting: study participants and recruitment
Pre-piloting was performed among a sample of the target 
population, beekeepers (n=14) of any age, registered with 
the Slovenian Beekeepers’ Association (SBA), according to 
which a beekeeper is a member of the SBA who undergone 
through a standardized educational and training system 
in the framework of a lifelong learning, led by the SBA, 
regardless beekeeping status (professional or amateur). 
This includes training under a supervisor (mentor) who 
is well educated in beekeeping, has long experience of 
beekeeping, uses the principles of good beekeeping 
practice, performs beekeeping nearby and in the same hive 
system, and is willing to share knowledge and experience 
in order to help the new beekeeper with the initial steps on 
their beekeeping path. The SBA includes 207 beekeeping 
societies and 14 regional beekeeping associations, with a 
total of about 7,500 amateur and professional beekeepers 
(19). The following exclusion criteria were applied: stings 
caused by Hymenoptera species other than bees (e.g., 
wasp, hornet); toxic reaction; acute or chronic conditions 
that prevent the beekeeper from participating in the 
study; refusal to participate after obtaining informed 
consent. The recruitment period took place from 25th to 
29th September 2021 using an online approach – a Google 
form published on the SBA website. Participants willing 
to take the survey marked their informed consent and 
expressed their preferred time and date to conduct the 
survey using a face-to-face method. The physical location 
for the in-person meetings was determined individually 
and tailored to the participant’s request.
2.2.4 Response process validation
For the quantitative assessment of the response process 
validation, the members of the rater panel were 
requested to rate the instrument items in terms of 
clarity and comprehensibility. The same scoring and the 
same recoding procedure were applied as for the CVIs. 
The item-level face validity index (I-FVI) was computed 
as the number of agreed items divided by the number of 
raters (Equation 4). Items with FVI values above 0.83 were 
considered to be accepted (20).
Equation 1
Equation 2
Equation 3 Equation 4
In relation to the number of experts, items and using the 
questionnaire as a tool, the cut-off scores of I-CVI ≥0.78 
(13) and S-CVI/Ave ≥0.90, respectively, were considered to 
be acceptable (16), while for S-CVI/UA a value of ≥0.80 was 
accepted as good (17, 18).

10.2478/sjph-2023-0019 Zdr Varst. 2023;62(3):137-144
140
2.3 Piloting 
The validated questionnaire was piloted among a group of 
beekeepers of any age, registered with the SBA, to assess 
its feasibility for the planned research approach. The same 
recruitment process and the same exclusion criteria were 
applied as for the response process validation with the 
data collection period from 15th to 29th October 2021. 
We included the first 50 beekeepers who filled out an 
online informed consent form, a number that was chosen 
based on a consensus among all authors of this paper. Due 
to the health risks posed by the COVID-19 pandemic and 
the associated restrictions, we were forced to switch our 
survey methodology from face-to-face to telephone. A 
telephone-to-paper method was used with the questions 
administered to the respondent during a phone call; the 
data were simultaneously recorded on a paper version of 
the questionnaire. The interviews were carried out by the 
same researcher with expertise in conducting telephone 
interviews. 
The Slovenian version of the questionnaire was translated 
into English using the forward-backward translation 
method. A copy of this version of the questionnaire is 
available from the first author.
2.4 Statistical analysis 
Descriptive statistics were used to describe the 
sociodemographic characteristics of the study participants, 
including the frequencies for categorical variables, and 
median values and their respective interquartile ranks 
(IQRs) for continuous variables. Data are shown in both 
graphic and tabular form. IBM SPSS (version 27) was used 
for the data analysis.
3 RESULTS 
3.1 Questionnaire development and validation 
3.1.1 Domain identification and item generation 
A synthesis of the data resulted in the identification 
of seven domains, as follows: 1) socio-demographics, 
2) beekeeping, 3) self-reported SAR to bee venom, 
4) treatment, 5) comorbidities, 6) smoking, and 7) 
beekeeping among family members. An initial pool of 46 
items was reviewed iteratively until the consensus group 
reached agreement on 39 items. The instrument was 
named APISS-Q, based on the genus name Apis for “bee”, 
the letter “S” for “Slovenia” and “Q” for “questionnaire”.
3.2 Validity assessment
3.2.1 Sociodemographic profile of the panel experts 
Of the 15 experts selected and invited to participate in 
the study, 13 responded to the mailed request (response 
rate: 86.7%). The panel experts were aged between 24 
and 66 years (mean age 46.7 years) with a slight female 
predominance (seven of 13; 53.9%). The vast majority 
came from the Central Slovenia statistical region (eight of 
13; 61.5%), followed by Gorenjska (two of 13; 15.4%), while 
Goriška, Savinjska, and Podravska statistical regions each 
had one participating expert (one of 13; 7.7%).
3.2.2 Sociodemographic profile of the pre-test study 
participants 
Out of 14 study participants, 11 were male and three were 
female, and they were aged between 34 and 71 years (mean 
age 54.7 years). The vast majority came from the Central 
Slovenia statistical region (six of 14; 42.9%), followed by 
the Goriška (three of 14; 21.4%) and Primorsko-notranjska 
(two of 14; 14.3%). Jugovzhodna, Gorenjska and Podravska 
statistical regions each had one pre-test study participant 
(one of 14; 7.1%).
3.2.3 Content validity and response process validation
The APISS-Q in its final form contained 50 items under the 
following domains: 1) socio-demography (i.e., gender, age, 
bodyweight, etc.), 2) beekeeping (i.e., type of beekeeping 
– amateur, professional; type of protective clothing while 
beekeeping, etc.), 3) self-reported SAR to bee venom 
(i.e., SAR grading system with the description of typical 
symptoms for each grade, the number of bees causing SAR, 
etc.), 4) treatment modalities of the self-reported SAR to 
bee venom (i.e., medical services accessed after the first 
SAR to bee venom, allergy specialist review after the first 
SAR to bee venom, etc.), 5) comorbidities and regular 
drug therapy (i.e., symptoms of atopy or other allergic 
diseases, other comorbidities, etc.), 6) behavioural risk 
factors (smoking status), and 7) beekeeping among family 
members/close blood relatives.
The relevance ratings of the 13 experts with regard to the 
item scale are shown in Table 1. All 50 items met the I-CVI 
threshold of ≥0.78. An average item quality expressed by 
the S-CVI/Ave was 0.97, indicating a high content validity 
of the questionnaire. The S-CVI/UA method, reflecting 
the proportion of items in the tool that achieved a 
rating of three or four by all the panel members, was 
below the value accepted as good (0.72). An item level 
of the response process validity index of 1.00 indicated 
clarity and comprehensibility for all items of the APISS-Q. 
The flow chart of the questionnaire development and 
validation process is shown in Figure 1.

10.2478/sjph-2023-0019 Zdr Varst. 2023;62(3):137-144
141
SOCIODEMOGRAPHY
BEEKEEPING
SELF-REPORTED SAR 
TO BEE VENOM
TREATMENT MODALITIES 
OF THE SELF-
REPORTED SAR TO 
BEE VENOM
COMORBIDITIES AND 
REGULAR DRUG 
THERAPY
BEHAVIOURAL 
RISK FACTORS
BEEKEEPING AMONG  
FAMILY MEMBERS/CLOSE 
BLOOD RELATIVES
Q1
Q2
Q3
Q4
Q5
Q6
Q7
Q8
Q9
Q10
Q11
Q12
Q13
Q14
Q15
Q16
Q17
Q18
Q19
Q20
Q21
Q22
Q23
Q24
Q25
Q26
Q27
Q28
Q29
Q30
Q31
Q32
Q33
Q34
Q35
Q36
Q37
Q38
Q39
Q40
Q41
Q42
Q43
Q44
Q45
Q46
Q47 
Q48
Q49
Q50
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
0.8
0.8
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
0.9
0.9
0.9
0.9
0.9
0.9
0.9
1.0
1.0
0.9
0.9
0.9
0.9
1.0
0.8
1.0
1.0
1.0
1.0
1.0 
1.0
1.0
1.0
∑48.6
13
13
13
13
13
13
13
13
13
13
13
11
10
13
13
13
13
13
13
13
13
13
13
13
13
13
13
12
12
12
12
12
12
12
13
13
12
12
12
12
13
11
13
13
13
13
13 
13
13
13
1
1
1
1
1
1
1
1
1
1
1
0
0
1
1
1
1
1
1
1
1
1
1
1
1
1
1
0
0
0
0
0
0
0
1
1
0
0
0
0
1
0
1
1
1
1
1 
1
1
1
∑36.0
ITEM DOMAINS I-CVI Expert in 
agreement
UA
Table 1.
Figure 1.
The relevance ratings of the 13 experts with regard to 
the item scale.
Flow chart of the questionnaire development and 
validation. Legend. APISS-Q – Apis for “bee”, the 
letter “S” for “Slovenia” and “Q” for “questionnaire”. 
Legend: I-CVI – item-level content validity index; S-CVI/Ave – 
scale-level content validity index based on the average method; 
S-CVI/UA – scale-level content validity index based on the 
universal agreement method; UA – universal agreement; SAR - 
systemic allergic reaction; Q – question. 
3.3 Piloting 
3.3.1 Demographic profile of the pilot study participants 
All beekeepers who filled out an online consent form 
responded to the survey (100% response rate). On 
average, it took a little more than 10 minutes to complete 
the telephone interview, with all participants finding the 
length to be acceptable. Out of 50 study participants, 43 
were male, and seven were female, aged between 18 and 
over 61 years (mean age 51.1 years). The vast majority 
came from the Central Slovenia statistical region (20 of 
50; 40.0%), followed by the Goriška (nine of 50; 18.0%); 
Gorenjska (eight of 50; 16.0%), Savinjska (seven of 50; 
14.0%), and Jugovzhodna (two of 50; 4.0%). Zasavska, 
Pomurska, Obalno-kraška and Primorsko-notranjska 
statistical region each presented had one study participant 
(one of 50; 2.0%). 
4 DISCUSSION 
The overriding aim of this study was to develop and validate 
a comprehensive questionnaire, making it possible, 
through the targeted environmental history, to collect 
epidemiological data about the self-reported SAR to bee 
venom and the risk factors for SAR among beekeepers, all 
vitally important for future clinical assessment in order to 
formulate a correct diagnosis and the related treatment, 
including counselling. Our findings have demonstrated 
the high calculated CVIs (I-CVI, S-CVI/Ave) and FVI (I-FVI) 
values, indicating the excellent validity of the tool. 
Subsequent piloting among beekeepers with no further 
adjustments suggests that this questionnaire is feasible 
for use for its intended purpose in public health and 
clinical settings. Translation into English may additionally 
contribute to its wider international use.
There have been few published studies in this field of 
science. To the best of our knowledge, the only available 
study questionnaire to date is the German one (QAHB), 

to distinguish between different clinical types of AR. 
Therefore, prior its use they should be well educated 
by a medical doctor. Next, switching questionnaire 
delivery mode during the study due to COVID-19 pandemic 
might have impacted the pilot phase. However, the 
epidemiological measures had to be respected and we 
firmly believe that this change did not have a major 
impact on the results. Nonetheless, the preferred mode 
for delivery is definitely face-to-face. Next, one of the 
major weaknesses of the scale used might be the lack 
of a “don’t know” answer. However, offering neutral 
options could allow experts and study participants in pre-
piloting to move on without giving careful thought to the 
domain and item investigated. In addition, apart from 
Yussof (14, 20), the use of a four-point Likert scale in the 
content validity was suggested (29) and used by several 
other authors (15, 30), and Chang (31), who compared 
four- and six-point Likert scales, reported that the former 
have greater reliability. Next, although several other 
consensus methods to evaluate interrater agreement for 
content validity purposes could be used (i.e., T, rWG, 
r*WG, content validity ratio (CVR)), the widely used CVI 
has advantages with regard to ease of computation, 
understandability, focus on agreement of relevance 
rather than agreement per se, focus on consensus rather 
than consistency, and provision of both item and scale 
information when compared it to alternative indexes (32). 
One weakness is its failure to adjust for chance agreement 
that can be in future solved by translating I-CVIs into 
values of a modified kappa statistic. Still, several recent 
studies (14, 20) established the content validity using the 
same CVI and FVI as we did to support the validity of an 
assessment tool. Next, a further shortcoming of the study 
is the value of S-CVI/UA, which was below the threshold of 
≥0.80, considered to show excellent content validity (33). 
Nonetheless, the UA calculation method was not used to 
evaluate the overall instrument as it may underestimate 
its content validity due to the reciprocal relationship 
between the probability that the experts would all 
agree on relevance and the number of experts (17). In 
other words, the higher the number of experts, the more 
difficult to achieve acceptable values for S-CVI/UA. Based 
on the literature, most studies avoid using the S-CVI/
UA approach, and thus a more flexible method for the 
overall instrument-CVI calculation is preferred (i.e., S-CVI/
Ave) (34). Next, it could be claimed that the differences 
between respondents and non-respondents regarding 
some characteristics (e.g. gender, age, education level 
and regional distribution) were not studied and shown. 
Unfortunately, due to the EU General Data Protection 
Regulation, the SBA could not provide us with a list 
of registered beekeepers with general demographic 
characteristics, hence we were forced to address the 
request for this to the presidents of the individual 
beekeeping societies. As a result, there were no data 
10.2478/sjph-2023-0019 Zdr Varst. 2023;62(3):137-144
142
provided by the author on our request (9). The QAHB 
was designed after gathering information from previous 
research among beekeepers and includes a large number 
of domains with some very detailed data. The whole 
instrument (QAHB and the Inventory for the Measurement 
of Bodily Negative Affectivity (INKA-h) questionnaire) was 
piloted in 10 volunteers prior to the survey to ensure 
its intelligibility. In addition, the authors included a 
reference group to rule out any potential biases between 
the responders and non-responders. However, the QAHB 
is rather long (13 pages), which may contribute to an 
experience of fatigue or a loss of motivation to complete 
it, albeit length per se is not the only measure of the effort 
needed to answer all the items. From the methodological 
point of view, the major limitation of the QABH lies in 
its development, as the questionnaire was no validated. 
The same is true for the questionnaires covering different 
types of allergies, as none of the questionnaires used were 
tested for content validity, using either systematic or non-
systematic approaches (21). This is important, as content 
validity, vital to support the validity of the assessment 
tool (14), in addition to the response process validity (20), 
are both needed to assess how far a survey instrument 
measures the intended idea, and thus is a high-quality 
assessment tool (22). In addition, with regard to the 
mode of delivery neither home interviewing nor personal 
delivery were used, both of which have been shown to be 
preferable methods of the tool distribution in terms of 
response rate (23). Taken together, the aforementioned 
factors may have contributed to the lowest response 
rate (3%) among the surveys included in our systematic 
literature review that provided the data. Among these 
studies, with the exception of Annila et al. (24), a rather 
low response rate (i.e., below 60%) has also been reported 
by a number of authors (5, 7, 25), and this has been found 
across all epidemiological designs in recent years (26). 
Despite not being recognized as evidence of study quality 
and validity alone, a low response rate may reduce the 
statistical power and prevent generalization to the wider 
community, thus representing an important component of 
the overall survey problems (27).
Our study has potential limitations that need to be 
addressed. Firstly, it could be argued that including both 
professional and amateur beekeepers is a weakness, since 
the latter are generally less educated about beekeeping. 
However, this is not the case in Slovenia, due to rigorous 
unified entry and the lifelong SBA educational and training 
system for all beekeepers, as explained in the methods 
section (19). Moreover, when developing the APISS-Q 
the specific characteristics of the Slovenian beekeeping 
were taken into account (28). Next, although data were 
collected by face-to-face and telephone interviews with 
a medical doctor, making it possible to clarify certain 
clinical concerns, we are aware that the questioners, if 
using the APISS-Q, might not have appropriate knowledge 

10.2478/sjph-2023-0019 Zdr Varst. 2023;62(3):137-144
143
available to compare respondents and non-respondents 
regarding their socio-demographic characteristics. 
However, given how the SBA takes care of the education 
of each new member, we felt that this would not adversely 
affect the survey results. Finally, we are aware that the 
detailed description/reporting of all specific questions 
within each domain and their refinement after the first 
draft would enhance the transparency and replicability 
of the study. Unfortunately, this was not possible due to 
journal limitations. However, all interested readers can 
get an insight into the questionnaire and its development 
stages in the Slovenian language from the first author 
upon request.
5 CONCLUSIONS
This may be the first study providing an evidence-based 
assessment tool and determining its validity using selected 
validity assessment indices. Numerous panel members 
with different fields of expertise, enabling the inclusion of 
all the relevant public health and clinical aspects across 
content domains, and the panel raters, randomly selected 
from the target group of the population, all reduce the 
effect of a chance agreement on the results. Piloting 
among beekeepers using a telephone approach suggested 
the tool to be feasible for its epidemiological use among 
beekeepers. The data gathered could have substantial 
benefits for clinicians, public health experts and 
policymakers that aim to mitigate the risk of SAR to bee 
venom among beekeepers and in the wider community.
CONFLICTS OF INTEREST
The authors have no competing financial interests or 
personal relationships to declare.
FUNDING
The work was supported by the Slovenian Research Agency 
[grant No. P3-0360, grant No. P3-0429]. 
ETHICAL APPROVAL
The study was approved by the National Medical Ethics 
Committee of the Republic of Slovenia (No. 0120-
423/2020-3, academic research). All participants gave 
written, informed consent prior to survey participation. 
AVAILABILITY OF DATA AND MATERIALS
The questionnaire and information on its development 
stages in Slovenian are available from the first author 
upon request. When the entire methodological process is 
fully completed, the English version of the questionnaire 
will also be available from the first author upon request.
REFERENCES
1. Bilo MB, Pravettoni V, Bignardi D, Bonadonna P, Mauro M, Novembre E, 
et al. Hymenoptera venom allergy: management of children and adults 
in clinical practice. J Investig Allergol Clin Immunol. 2019;29(3):180-
205. doi: 10.18176/jiaci.0310.
2. Casale TB, Burks AW. Clinical practice. Hymenoptera-sting 
hypersensitivity. N Eng J Med. 2014;10;370(15):1432-1439. doi: 
10.1056/NEJMcp1302681.
3. Bilo MB, Tontini C, Martini M, Corsi A, Agolini S, Antonicelli L. Clinical 
aspects of Hymenoptera venom allergy and venom immunotherapy. 
Eur Ann Allergy Clin Immunol. 2019;51(6):244-258. doi: 10.23822/
EurAnnACI.1764-1489.113.
4. Blank S, Haemmerle S, Jaeger T, Russkamp D, Ring J, Schmidt-Weber 
CB, et al. Prevalence of Hymenoptera venom allergy and sensitization 
in the population-representative German KORA cohort. Allergo J Int. 
2019;28(6):183-191. doi: 10.1007/s40629-018-0089-4.
5. Becerril-Angeles M, Nuñez-Velázquez M, Grupo del Programa de 
Control de la Abeja Africanizada, SAGARPA. [Risk factors for allergy 
to honey-bee venom in Mexican beekeepers]. Rev Alerg Mex. 
2013;60(3):100-104. 
6. Ediger D, Terzioglu K, Ozturk RT . Venom allergy, risk factors for systemic 
reactions and the knowledge levels among Turkish beekeepers. Asia 
Pac Allergy. 2018;8(2):e15. doi: 10.5415/apallergy.2018.8.e15.
7. Richter AG, Nightingale P, Huissoon AP, Krishna MT. Risk factors for 
systemic reactions to a bee venom in British beekeepers. Ann Allergy 
Asthma Immunol. 2011;106(2):159-163. doi: 10.1016/j.anai.2010.11.005.
8. Carli T, Locatelli I, Košnik M, Kukec A. Epidemiology and risk factors 
of self-reported systemic allergic reactions to a Hymenoptera venom 
in beekeepers: A protocol for a systematic literature review of 
epidemiological observational studies in beekeepers worldwide. BMJ 
Open. 2022;12(7):1-5. doi: 10.1136/bmjopen-2021-058130.
9. Münstedt K, Hellner M, Winter D, von Georgi R. Allergy to bee 
venom in beekeepers in Germany. J Investig Allergol Clin Immunol. 
2008;18(2):100-105. 
10. Prattala R, Helasoja V, Laaksonen M, Laatikainen T, Nikander P, Puska 
P. CINDI health monitor. Proposal for practical guidelines. Helsinki: 
Publications of the National Public Health Institute [Internet]. 2019 
[cited 2023 Apr 27]. Available from: https://www.julkari.fi/bitstream/
handle/10024/78046/2001b14.pdf;sequence=1
11. Fernandez-Gomez E, Martin-Salvador A, Luque-Vara T, Sanchez-Ojeda 
MA, Navarro-Prado S, Enrique-Miron C. Content validation through 
expert judgement of an instrument on the nutritional knowledge, 
beliefs, and habits of pregnant women. Nutrients. 2020;12(4):1136. 
doi: 10.3390/nu12041136.
12. McGartland Rubio D, Berg-Weger M, Tebb SS, Lee ES, Rauch S. 
Objectifying content validity: Conducting a content validity study in 
social work research. Soc Work Res. 2003;27(2):94-104. doi: 10.1093/
swr/27.2.94.
13. Davis KA. Validity and reliability in qualitative research on second 
language: Acquisition and teaching. Another Researcher Comments. 
TESOL Q. 1992;26(3):605-608. doi: 10.2307/3587190.
14. Yusoff MSB. ABC of content validation and content validity index 
calculation. Med Educ. 2019;11(2):49-54. doi: 10.21315/eimj2019.11.2.6.

10.2478/sjph-2023-0019 Zdr Varst. 2023;62(3):137-144
144
15. Zamanzadeh V, Ghahramanian A, Rassouli M, Abbaszadeh A, Alavi-
Majd H, Nikanfar A-R. Design and implementation content validity 
study: Development of an instrument for measuring patient-centered 
communication. J Caring Sci. 2015;4(2):165-178. doi: 10.15171/
jcs.2015.017.
16. Polit DF, Beck CT. The content validity index: are you sure you know 
what’s being reported? Critique and recommendations. Res Nurs 
Health. 2006;29(5):489-497. doi: 10.1002/nur.20147.
17. Grant JS, Davis LL. Selection and use of content experts for instrument 
development. Res Nurs Health. 1997;20(3):269-274. doi: 10.1002/
(sici)1098-240x(199706)20:3<269:aid-nur9>3.0.co;2-g.
18. Hambleton RK, Swaminathan H, Algina J, Coulson D. Criterion-
referenced testing and measurement: A review of technical 
issues and developments. Rev Educ Res. 1978;48:1-47. doi: 
10.3102/00346543048001001.
19. Slovenian Beekeepers Association. General information [Internet]. 
[cited 2023 Mar 13]. Available from: https://en.czs.si/about/general-
information/
20. Yusoff MSB. ABC of response process validation and face validity index 
calculation. Med Educ. 2019;11(3):55-61. doi: 10.21315/eimj2019.11.3.6.
21. Sin BE, Öztuna D, Gelincik A, Gürlek F, Baysan A, Sin AZ, et al. Quality-
of-life in insect venom allergy: Validation of the Turkish version of the 
“Vespid Allergy Quality of Life Questionnaire” (VQLQ-T). Springerplus. 
2016;5:583. doi: 10.1186/s40064-016-2246-x. 
22. Elangovan N, Sundaravel E. Method of preparing a document for 
survey instrument validation by expertes. MethodsX. 2021;8:101326. 
doi: 10.1016/j.mex.2021.101326.
23. Rolstad S, Adler J, Rydén A. Respone burden and questionnaire 
length: Is shorter better? A review and meta-analysis. Value Health. 
2011;14(8):1101-1108. doi: 10.1016/j.jval.2011.06.003.
24. Annila IT, Karjalainen ES, Annila PA, Kuusisto PA. Bee and wasp 
sting reactions in current beekeepers. Ann Alergy Asthma Immunol. 
1996;77(5):423-427. doi: 10.1016/S1081-1206(10)63342-X.
25. Celikel S, Karakaya G, Yurtsever N, Sorkun K, Falyoncu AF. Bee 
and bee products allergy in Turkish beekeepers: determination of 
risk factors for systemic reactions. Allergol Immunopathol (Madr.). 
2006;34(5):180-184. doi: 10.1157/13094024.
26. Morton LM, Cahill J, Hartge P . Reporting participation in epidemiologic 
studies: A survey of practice. Am J Epidemiol. 2005;163(3):197-203. 
doi: 10.1093/aje/kwj036.
27. Blumenberg C, Menezes AMB, Gonçalves H, Formoso Assunçao MC, 
Wehrmeister FC, Barros FC, et al. The role of questionnaire length and 
reminders frequency on response rates to a web-based epidemiologic 
study: A randomised trial. Int J Soc Res Methodol. 2019;22(6):626-635. 
doi: 10.1080/13645579.2019.1629755.
28. Beekeeping Academy of Slovenia. Beekeeping in Slovenia [Internet]. 
2012 [Cited 2023 Mar 17]. Available from: https://sca.kis.si/en/
slovenian-beekeeping/
29. Lynn MR. Determination and quantification of content validity. Nurs 
Res. 1986;35(6):382-385.
30. Jokiniemi K, Meretoja R, Pietilä AM. Constructing content validity of 
clinical nurse specialist core competencies: Exploratory sequential 
mixed-method study. Scand J Caring Sci. 2018;32(4):1428-1436. doi: 
10.1111/scs.12588.
31. Chang L. A psychometric evaluation of 4-point and 6-point Likert-
type scales in relation to reliability and validity. Appl Psychol Meas; 
1994:18(3):205-215. doi: 10.1177/01466216940180030.
32. Polit DF, Tatano Beck C, Owen SVV. Is the CVI an acceptable indicator 
of content validity? Appraisal and recommendations. Res Nurs Health. 
2007;30(4):459-467. doi: 10.1002/nur.20199.
33. Rodrigues IB, Adachi JD, Beattle KA, MacDermid JC. Development 
and validation of a new tool to measure the facilitators, barriers 
and preferences to exercise in people with osteoporosis. BMC 
Musculoskelet Disord. 2017;18(1):540. doi: 10.1186/s12891-017-1914-5. 
34. Almanasreh E, Moles R, Chen TF. Evaluation of methods used for 
estimating content validity. Resocial Adm Phar. 2019;15(2):214-221. 
doi: 10.1016/j.sapharm.2018.03.066.