Guidelines for designing a research project
This document specifies a series of tasks and activities that students
or researchers should complete to optimise their research projects.
Although relevant to researchers, the document is primarily directed to
fourth year and postgraduate psychology students. These principles
should be regarded merely as guidelines, not inflexible rules.
Part 1 - Clarify your interests
Task 1. Have you collected enough information from previous literature?
According to recent research, the most effective, progressive, and
informative projects integrate ideas and constructs from several
related, but distinct, topics. In other words, you should:
- Initially, glean a series of interesting arguments and findings from the literature.
- Then, attempt to integrate these ideas and findings into a unified framework.
Typically, not all of these ideas and findings can be integrated.
Nevertheless, attempts to integrate many diverse concepts tend to
stimulate innovation and thus progress. Therefore, before you finalise
your topic, you should collect a broad gamut of ideas and fact. For
example:
- You should collate suggestions for future research from articles you perceive as interesting.
- In addition, you should uncover some counterintuitive, illuminating
empirical findings in these fields. That is, you should collect
findings in which the opposite outcome would have been equally, if not
more, plausible. To illustrate, some studies have revealed that
pressing up on a table enhances creativity, a finding that is clearly
unexpected rather than inevitable.
- Finally, you should collect the explanations that were utilised to accommodate these unexpected findings.
-
You should, however, focus on explanations that are theoretical. In
other words, you should collect explanations that could apply to many
contexts and settings. You should not focus on methodological
explanations that seem specific to one study but do not apply to other
contexts, settings, or approaches.
-
In summary, you should:
- Collect between 5 and 10 suggestions for future research in literature that corresponds or relates to your interests
- Collect between 10 and 20 counterintuitive, illuminating empirical findings.
- Collect theoretical explanations that were proposed to reconcile these findings.
Task 2. Have you specified a range of independent and dependent variables?
Once you have completed the first task, you should consider the
independent and dependent variables that you would like to explore. For
example, you might want to examine the effect of body odour on the
extent to which individuals are perceived as attractive by onlookers.
In this context, body odour can be regarded as the independent
variable, or cause, and attraction can be regarded as the dependent
variable, or outcome.
You could select independent and dependent variables that have either:
- Seldom--if ever--been studied simultaneously, but nevertheless could be related.
- Often been studied simultaneously, but have spawned conflicting findings.
Often been studied simultaneously, but the sequence of events that
underpin their association has not been established definitively.
The number of independent and dependent variables varies appreciably
across research projects. Most large theses, such as doctoral projects,
comprise between 2 and 5 sets of independent variables. For example,
the independent variables might include personality, which might
comprise 5 distinct traits, and coping, which might comprise 3 distinct
styles. The total number of independent variables might thus range from
6 to 30 depending on the number of subscales associated with each set.
Most large theses comprise between 1 and 4 dependent variables, such as
self esteem and wellbeing.
Most smaller theses, such as fourth year projects, comprise between 1
and 3 sets of independent variables. The total number of independent
variables might thus range from 4 to 18. These projects also comprise
between 1 and 3 dependent variables.
In summary, you should:
- Ensure the study entails an appropriate number of primary independent and dependent variables.
- Specify the independent and dependent variables you plan to explore in the space provided.
Task 3. Have you considered variables that might mediate these relationships?
Suppose you were interested in whether or not body odour influences the
extent to which individuals are perceived as attractive. In addition,
suppose a series of studies have shown that individuals seem more
attractive after smothered in dencorub. This finding, alone, however is
not informative. In particular:
- This finding does not ascertain why dencorub enhances attraction.
- As a consequence, we cannot determine whether or not this finding would persist in other settings.
- Indeed, we cannot even determine whether or not this finding would persist in the same setting tomorrow.
Somebody might retort, "The day of the week should not influence this
finding. Therefore, this finding should persist tomorrow". This retort,
however, assumes that day of the week is irrelevant. This assumption,
however, assumes some understanding of why dencorub enhances
attraction--an understanding that cannot be gleaned from the finding
alone. The optimal method to develop this understanding is to explore
the sequence of events that intervenes between the independent and
dependent variable.
First, you need to uncover possible explanations or mechanisms that
could intervene between the independent and dependent variable.
In this example, perhaps when individuals apply dencorub, they are perceived by onlookers as active and thus attractive.
Second, you need to identify the mediators this explanation entails.
In this example, the explanation implies that perceived level of
activity mediates the relationship between dencorub and attraction, as
illustrated below.
Finally, you need to either measure or control this mediator. For
instance, you could create a scale that measures the extent to which
participants are perceived as active. Alternatively, you could control
this mediator methodologically. The individuals who are rated might be
athletes only, whose level of activity will be perceived as high. If
this variable is indeed a mediator, controlling that variable, either
statistically or methodologically, should eliminate the relationship
between the independent and dependent variables. This finding would
thus provide an insight into the pathway that links the independent and
dependent variables.
In summary, you should, if possible:
- Identify several explanations that describe why the independent variables influence the dependent variables
- Each explanation implies the independent and dependent variables will
be mediated by specific variables. These variables are called
mediators..
Part 2 - Assessing alternative explanations
Task 4. Have you considered spurious variables that influence both your independent and dependent variables?
Suppose you discover that your independent variables correlate with
your dependent variables. For example, individuals who wear dencorub
might be perceived as more attractive. This finding does not
necessarily indicate that dencorub enhances attraction. Instead, a
spurious variable, which is a factor that influences both the
independent and dependent variable, might underpin this finding. This
concept is illustrated below.
To illustrate, suppose that young individuals are more likely to both
apply dencorub and appear attractive. However, suppose that dencorub
itself does not influence attraction. The table below provides some
hypothetical results. The first column specifies the age group of
participants. The second column specifies the number of times each year
they apply dencorub. The third column specifies the extent to which
they are perceived as attractive, on a scale from 0 to 20. The top set
of rows comprises young individuals, who will often tend to apply
dencorub and seem attractive. The bottom set of rows comprises older
individuals, who seldom apply dencorub or seem attractive.
Taken together, these findings suggest that dencorub correlates with
level of attraction. That is, individuals who apply dencorub tend to be
perceived as more attractive. Nevertheless, when each age group is
considered separately, dencorub does not correlate with level of
attraction. Thus, to understand the effect of dencorub on attraction,
age should be controlled, either statistically or methodologically.
Alternatively, participants should be randomly assigned to various
conditions, called an experimental design, which tends to ensure these
spurious variables are controlled. In short, spurious variables should
ideally be controlled. Spurious variables that cannot be controlled
should be conceded in the limitations section of your thesis.
In summary, if you do not plan to randomly assign participants to conditions, you should:
- Identify variables the
literature suggests could influence one or more of your dependent
variables, apart from the independent variables and mediators you
specified earlier
- Identify which of these variables could also influence one or more of
your independent variables or mediators. Only consider the independent
variables that you are not manipulating yourself.
- Variables that influence both your independent and dependent variables are designated as spurious.
Task
5. Have you considered whether or not relationships between your
independent and dependent variables could be ascribed to common method
variance?
A prevalent, almost ubiquitous, variant of a spurious variable is
called common method variance. Specifically, each participant
demonstrates a particular tendency or strategy when completing surveys
or other measures. For example, some participants always respond
favourably or leniently. In contrast, some participants always provide
neutral responses. Nevertheless, some participants always respond
unfavourably or harshly. The following table presents some hypothetical
data. In this example, the first three rows represent lenient
participants, the next three rows represent neutral participants, and
the final three rows represent harsh participants. Again, the two
variables appear to be related& high scores on one variable correspond
to high scores on the other. This relationship, however, is spurious
and merely reflects the observation that each participant applies the
same strategy to each question--a problem denoted as common method
variance.
Several strategies can be undertaken to minimize common method variance. These strategies are described by Podsakoff, MacKenzie, Lee, and Podsakoff (2003). Specifically:
First, common method variance diminishes if the questions are
concrete and objective. For example, a question such as, "How often do
you attend parties alone--once a week, month, year, or less than once a
year" is relatively objective. Spies who had watched these participants
over many years would provide the same responses as the participants
themselves& hence, these responses are objective.
Second, if possible, the questions should refer to current, rather
than retrospective, attitudes, perceptions, or behaviours.
Retrospective questions, such as "At primary school, did you like the
smell of dencorub", are more susceptible to various biases such as
attempts to maintain consistency across responses.
Third, you could apply different response scales to measure each
variable. A series of 'Yes' and 'No' scales could be utilised to gauge
the dependent variables. A series of Likert scales could be utilised to
gauge the independent variables.
Fourth, you could measure the variables from the perspective of
different individuals. For example, if employees complete the scales
that gauge the independent variables, their managers could complete the
scales that gauge the dependent variables.
Fifth, you could vary the time and location at which each variable is
measured. For example, the independent variables could be measured one
day after the dependent variables.
Sixth, you should ensure the scales seem superficially dissimilar.
Seventh, you could encourage individuals to respond honestly. You
could introduce procedures that emphasise the anonymity of these
measures or express statements such as, "There are no right and wrong
answers" or "Please ensure your responses are accurate rather than
lenient or harsh". These tactics reduce the likelihood that
participants will exhibit leniency towards themselves or individuals
they like.
Eighth, you could measure positive affect, negative affect, and mood
in participants, perhaps using the PANAS or POMS. You could then
include these measures as control variables. In other words, these
measures could be designated as covariates - in the context of ANOVAs -
or additional predictors - in the context of regression analyses.
Hence, affect and mood - which can bias responses - is controlled.
Ninth, you could apply other sophisticated statistical techniques.
For example, some researchers undertake an exploratory factor analysis,
save factor scores, and control the first factor in all subsequent
analyses. Other researchers apply techniques called MTMM or correlated
uniqueness models to control common method variance.
In summary, you should:
Identify whether or not the common method variance could contaminate your findings.
This problem is rife if the measures that are applied to gauge the
independent, mediator, and dependent variables utilise a similar
format. For example, do measures of both the independent and dependent
variables entail a Likert scale in which participants specify their
level of agreement to a series of statements? If so, common method
variance is likely.
This problem is also rife if the questions are particularly
subjective and abstract or the answers on one scale might influence the
responses to a subsequent scale.
Consider strategies to reduce common method variance. For example,
you might need to adapt the items of established scales to ensure they
are more concrete and objective, but not retrospective. You could
utilise different individuals, formats, times, or locations to measure
the independent and dependent variables. Finally, you could apply
statistical procedures to nullify the effect of common method variance.
Task
6. Have you considered suppressor variables that might obscure
relationships between your independent and dependent variables?
Sometimes, a suppressor variable might obscure the relationship between
an independent and dependent variable. To illustrate, consider the
model below. According to this model, dencorub directly reduces
attraction, perhaps because the odour is offensive. On the other hand,
dencorub facilitates fitness, which in turn enhances attraction. These
two pathways nullify one another.
As a consequence, the correlation between dencorub and attraction will
not reach significance. Nevertheless, if fitness is controlled, perhaps
through the inclusion of this variable in a regression, the bottom
pathway is obstructed. Hence, the top pathway will prevail. That is,
dencorub will be inversely related to attraction. In other words,
relationships that become significant only after some variable is
controlled confers vital information. This outcome can arise if :
The independent and dependent variables are negatively related to one
another, but a mediator or spurious factor is positively related to
both variables or negatively related to both factors, as depicted in
the previous diagram.
The independent and dependent variables are positively related to one
another, but a mediator or spurious factor is positively related to one
of these variables and negatively related to the other. In each
instance, the top and bottom pathways nullify one another
In summary, you should:
If the independent and dependent
variables are positively related, identify mediators or spurious
variables that you uncovered earlier that positively correlate with the
independent variables but negatively correlate with the dependent
variable, or vice versa. These variables are potential suppressors.
If the independent and dependent variables are negatively related,
identify mediators or spurious variables that you uncovered earlier
that positively correlate with both the independent and dependent
variables or negatively correlate with both the independent and
dependent variables. These variables are also potential suppressors.
Specify the suppressors you plan to measure or control as well as the suppressors you do not plan to measure or control.
Task
7. Have you considered the possibility that perhaps the direction of
causation in relation to your independent, mediator, and dependent
variables is the opposite to your hypotheses?
Suppose the researcher hypothesises that dencorub enhances attraction,
perhaps because this product fosters fitness. Furthermore, suppose that
elevated use of dencorub does indeed correlate with attraction.
Contrary to this hypothesis, this finding might arise because level of
attraction might influence application of dencorub. Specifically,
unattractive individuals might become unconfident and thus decide to
refrain from dencorub to enhance their popularity. In other words, the
direction of causality might oppose the hypothesis. Accordingly the
finding does not support the theory that dencorub fosters fitness and
thus attraction.
Researchers can undertake a variety of measures to discredit this
alternative direction of causality. First, they could identify and
assess mediators that are compatible with one direction but not the
other. For example, suppose that fitness level - but not confidence
mediates - the relationship between dencorub and attraction. This
finding would support the hypothesis that dencorub enhances attraction
rather than vice versa.
Likewise, researchers could identify and assess moderators - that is
variables that influence the magnitude and direction of some
relationship - that are compatible with one direction but not the
other. For instance:
Suppose that justification of use moderates the relationship between dencorub and attraction.
In particular, suppose the relationship between dencorub and
attraction diminishes in participants who utilise dencorub merely
because the odour fosters relaxation.
In these participants, dencorub is unlikely to enhance fitness.
Accordingly, the finding that justification of use moderates the
relationship between dencorub and attraction is compatible with one
direction only.
In particular, this finding is compatible with the hypothesis that dencorub facilitates fitness and thus attraction.
Alternatively, researchers can refine the research design to
distinguish between the two directions. First, researchers could
manipulate the independent variable--that is, randomly assign
participants to each condition, called an experimental design. To
illustrate, half the participants could be asked to apply dencorub and
the remaining participants could be asked to refrain from dencorub.
Suppose the application of dencorub still correlates with attraction.
Clearly, this finding cannot be ascribed to the hypothesis that
attraction influences the use of dencorub. Finally, a longitudinal
design could be considered. That is:
Suppose that dencorub at time 1 correlates with attraction at time 2 after controlling attraction at time 1.
Nevertheless, suppose that attraction at time 1 does not correlate
with dencorub at time 2 after controlling dencorub at time 1.
This pattern of findings supports the hypothesis that dencorub influences attraction and not vice versa.
This argument invokes the rationale that causes tend to precede effects.
In summary, you should:
Identify several explanations that describe why the dependent variable could influence the independent variable.
These explanations could suggest the possibility of other mediators.
These explanations could also suggest the possibility of other
moderators--variables that influence the magnitude and direction of the
relationship between the independent and dependent variables.
If these mediators and moderators are compatible with both directions
of causality, consider another avenue to assess the direction of
causality. Either manipulate the independent variables or consider a
longitudinal design.
Part 3 - Generalizing the findings
Task
8. Do you believe the relationships between the independent, mediator,
and dependent variables will generalise to other contexts and
populations
Most theories are intended to apply to a particular population or
context. For example, the theory that dencorub enhances fitness and
thus attraction is probably intended to apply to adults - and not
children. Furthermore, this theory is probably intended to apply both
inside and outside buildings. Admittedly, these boundaries are implicit
rather than explicit, but are nevertheless obvious. Of course, your
sample and methodology will not necessarily be representative of this
entire population and context. That is:
Your study is unlikely to be representative of all adults, but merely
representative, or even unrepresentative, of adults in one city or
community.
Your study is unlikely to be representative of all contexts. For
example, your study might be conducted inside a University building.
Researchers should determine whether or not their sample and
methodology differs from the population and context to which this
theory should apply on several variables. For example, perhaps the
adults on this sample are less familiar with rank odours than is the
population. Alternatively, perhaps the odour of dencorub inside a
University building is less salient that is the odour of denorub in
most contexts. In other words, of course, the findings of this study
might not apply to all relevant populations and contexts. To overcome
this problem, researchers can pursue two possible options. First:
Researchers can ensure their sample and context is representative on these variables.
In this instance, the researcher would need to ensure the
distribution of 'familiarity with rank odours' in the sample mirrors
the population.
Likewise, the researcher would need to ensure the distribution of
'odour salience' in this study also mirrors the range of possible of
contexts.
Unfortunately, this option is often impractical.
Second, the researcher could ensure these variables, familiarity
with rank odours and odour salience, vary to some extent across
participants. The researcher could then measure these variables.
Finally, the researcher could assess whether or not the relationships
between the independent, mediator, and dependent variables are
moderated by these variables. This approach, called a moderated model
and depicted below, determines the extent to which the relationships
between the independent, mediator, and dependent variables apply to a
broad range of individuals and contexts.
In summary, you should:
Identify the contexts and populations in which the theoretical
explanations you would like to assess should apply, such as all
non-clinical settings, parents, and so on.
Identify the context and populations to which you have restricted your sample, such as large organisations and mothers
Consider variables differ between the context of your study and the
breadth of contexts in which the theoretical explanations apply.
Are any of these variables likely to moderate the relationships between your independent, mediator, and dependent variables.
Can you measure or vary these variables and thus examine whether or
not they moderate the relationships between your independent, mediator,
and dependent variables.
Part 4 - Preventing confounded treatments
For between-subject manipulations only. If your study does not comprise
any between-subject manipulations, in which each condition comprises
separate participants, proceed to Task 11.
Task
9. Have you considered whether or not compensation, demoralisation,
demand characteristics, Hawthorne effects, or some other confounding
variable could explain the observed relationships between independent
and dependent variables.
Between-subject treatments, in which each condition comprises separate
participants, can present several drawbacks. Specifically, these
treatments might not manipulate only the independent variable. Instead,
these treatments might also manipulate some other confounding variable.
This confounding variable could, in turn, influence the dependent
variable. Thus, treatments that affect the dependent variable cannot
necessarily be ascribed to the independent variable. This sequence of
events is illustrated in the figure below.
To illustrate some common confounding variables, suppose some
participants are asked to apply dencorub, whereas other participants
are asked to refrain from the application of dencorub. Participants
asked to refrain from dencorub might feel unfit. To compensate, they
might undertake more exercise, which in turn could influence the
dependent variable, the extent to which they are perceived as
attractive. Although this example might seem contrived, compensation
could explain the effect of some manipulation, especially if the
participants who receive the treatment could feel complacent (Campbell
& Stanley, 1963).
Nevertheless, the participants who do not receive dencorub might feel
demoralised. For instance, they might feel upset they had not received
the treatment. As a consequence, their motivation wanes or their
frustration heightens, which in turn could influence the dependent
variable (Campbell & Stanley, 1963).
Along similar lines, participants might decipher the hypotheses.
Participants who receive the dencorub might believe they are expected
to be fit and attractive, and this belief could impinge upon their
behaviour. Likewise participants who do not receive the dencorub might
believe they are expected to be unfit and unattractive. This effect of
perceived expectations on performance is often denoted as demand
characteristics.
Finally, novel experiences tend to foster hope and motivation.
Accordingly, the application of dencorub might promote this sense of
inspiration, which in turn could influence the dependent variable--a
sequence of events referred to as the Hawthorne effect. This sense of
inspiration might not persist, however. As a consequence, the finding
that dencorub enhances inspiration and thus attraction might represent
only a transient effect.
In summary, you should:
Determine whether or not participants in one condition might feel
more complacent or unmotivated than participants in another condition.
Identify measures that you have considered to minimise this complacency
if needed
Determine whether or not participants might decipher the hypotheses.
In the space provided, specify measures that you have considered to
obscure the hypotheses, if necessary. Alternatively, justify why their
recognition of these hypotheses would not influence their scores on the
dependent variables.
Determine whether or not one condition involves a more pronounced
sense of novelty than another. Identify measures that you have
considered to equate this sense of novelty& otherwise, you will need to
concede this shortcoming in the limitations section.
Task
10. Could experimenters vary the inadvertent cues they exhibit, or the
procedures they use to measure variables, across the conditions.
Sometimes, experimenters attempt to conceal the treatment. For
instance, consider the previous study in which only half the
participants receive the dencorub. Perhaps, to conceal this
manipulation:
Some product is applied to all participants.
Half the participants receive dencorub.
The other participants receive some other product that confers no effect.
Participants wear a gasmask to ensure they cannot decipher the difference.
This approach would preclude many confounding variables.
Nevertheless, the researcher might be aware of which participants
receive the treatment. For example, they might exhibit an expression of
disgust only when they apply dencorub. Participants might decipher this
subtle hint, which could influence their behaviour. Rather than exhibit
these subtle hints, researchers who are aware of which participants
receive the treatment might not measure attraction fairly. In other
words, this awareness could somehow bias the measure. To illustrate:
Suppose the researchers ask onlookers to verbalise whether or not each participant seems attractive.
The researcher who is aware that some participant received the
treatment, and thus assumes this individual will be perceived as
attractive, might misconstrue the response of onlookers.
Again, although this example is contrived, the possibility that biases could influence subjective measures is genuine.
In summary, you should:
Identify measures you will
introduce to ensure that participants do not decipher the treatment
they have received from the behaviour of experimenters. Perhaps these
experimenters who interact with participants are unaware of which
treatment each individual receives, which is called a double-blind
procedure.
In addition, specify tactics you will introduce to ensure that
measures of the dependent variable are not unbiased. This problem is
confined to observational data.
Task
11. If the conditions are not counterbalanced, have you considered
whether or not time influences the dependent variables (see Between-subject versus repeated-measures designs).
If your study does not comprise any within-subject or repeated-measures
manipulations, in which each condition utilises the same participants,
proceed to Task 14. Within-subject treatments, in which each condition
utilises the same participants, can also be contaminated by confounding
variables. For example:
Suppose that onlookers rate the extent to which they perceive participants as attractive on two occasions.
First, onlookers rate the participants before they receive dencorub.
Second, they rate the participants after they receive dencorub.
Suppose that participants are rated as less attractive after they
receive dencorub. This finding might not indicate that dencorub
tarnishes the appearance of participants. Instead, this finding could
be ascribed to other factors that vary across time.
Perhaps the participants aged marginally across time.
Perhaps the participants began to fatigue.
Counterbalancing circumvents this drawback (see Between-subject versus repeated-measures designs).
To counterbalance the conditions in this example, onlookers could rate
the extent to which they perceive participants as attractive on two
days. On the first day, only half the participants will receive
dencorub. The remaining participants receive no dencorub. The
participants who received dencorub on the previous day would receive no
dencorub on the second day, and vice versa, as depicted in the table
below.
This approach, however, cannot be applied when the design comprises one
participant only, often called single-subject designs. In these
instances, the researcher must select one of three options. First, an
ABAB design could be applied:
That is, the researcher could repeat the treatment and control conditions on many occasions.
To illustrate, on every second day they could ask participants to apply dencorub.
On every other day, they could ask participants to refrain from dencorub.
Hence, any effects of dencorub cannot be ascribed to time.
Unfortunately, to undertake this approach, researchers must withdraw
the treatment during every second session, which is sometimes
unethical. Hence, a multiple baseline design could be applied instead.
That is:
- The researcher could apply two treatments, which are hypothesised to influence separate dependent measures
- To illustrate, on the first day, no treatment is applied
- On the second day, dencorub, which is hypothesised to enhance attraction, is applied
- On the third day, both dencorub and vegemite, which is hypothesised to enhance mood, is applied
- If the hypotheses are supported, attraction should be elevated on the
second and third day. Mood should be elevated on the third day only.
- This rationale can be extended to more than two treatments
Unfortunately, this approach can be applied only when the researcher
wants to explore several treatments that influence distinct outcomes.
If this condition is not fulfilled, an incremental approach can be
adopted instead. For example:
- On the first day, no treatment is applied
- On the second day, 5 mg of dencorub is applied
- On the third day, 10 mg of dencorub is applied
- On the fourth day, 15 mg of dencorub is applied
- If the hypothesis is true, attraction should increase gradually over time.
These single-subject designs can be applied when the project comprises
more than one participant. For example, this design is applicable when
the project comprises five participants, but each individual
effectively represents a separate study.
In summary, you should:
- Determine whether or not scores
on the dependent variable could vary across time - perhaps as a
consequence of practice, fatigue, or aging. Justify your response in
the space provided.
- If the dependent variable could vary across time, you should
counterbalance the order in which the conditions are presented.
Alternatively, you should somehow minimise the effect of practice,
fatigue, an aging on the dependent variables.
Task 12. If participants withdraw, have you considered the possibility that later conditions might be biased?
If the conditions are not counterbalanced, another factor that could
vary across time is the response rate. For example, suppose again that
onlookers rate the extent to which they perceive participants as
attractive on two occasions: before and after the application of
dencorub. After the first set of ratings, some of the participants
might withdraw from the study. These participants might have received
unfavourable ratings, and thus might have withdrawn because their
confidence had been decimated. As a consequence, the individuals who
participated in the second condition thus tended to be more attractive.
The finding that participants in the second condition were more
attractive than were participants in the first condition cannot
necessarily be attributed to dencorub. Instead, this finding might
arise because individuals in the second condition did not represent an
unbiased, representative sample.
In summary, you should:
- Estimate the number of participants who might withdraw during the study and thus complete only some of the conditions.
- Determine whether or not these participants are likely to yield especially high or low scores on the dependent variables.
- If so, you will need to counterbalance the conditions. This approach
will be effective, provided the likelihood of withdrawal does not
depend on the order in which participants complete the conditions.
Task 14. If the conditions are counterbalanced, have you considered the possibility of asymmetric transfer
Counterbalancing, unfortunately, does not eradicate all confounding
variables. Instead, a subtle but consequential problem often arises,
typically called asymmetric transfer. Consider the method that is
represented in the table below.
First day | Second day
|
First group of participants | Receive dencorub | No dencorub
|
Second group of participants | No dencorub | Receive dencorub |
To explicate asymmetric transfer, consider the participants who applied
dencorub on the first day but not the second. The application of
dencorub on the first day could alleviate any injuries that
participants had recently endured. On the second day, these
participants thus feel healthier and thus more attractive. In other
words, participation in the condition that involves the application of
dencorub might subsequently enhance performance in the condition that
involves no dencorub, as represented by the asterisks in the table
below.
First day | Second day
|
First group of participants | Receive dencorub | No dencorub ****
|
Second group of participants | No dencorub | Receive dencorub |
In contrast, consider the participants who did not apply dencorub on
the first day, but did apply dencorub on the second day. Participation
in the condition that involves no dencorub will not subsequently affect
performance in the condition that involves dencorub. Taken together, in
the first group of participants, performance in the condition that
involves no dencorub will receive an unfair benefit: a problem referred
to as asymmetric transfer.
In summary, you should:
- Estimate the extent to which participation in Condition A could enhance or impair performance in Condition B
- Estimate the extent to which participation in Condition B could
enhance or impair performance in Condition A. Extend this rationale if
your design entails more than 2 conditions.
- If asymmetric transfer is likely, consider a between-subject design instead.
Part 5 - Optimizing measures
Task 14. Have you collected evidence to suggest that your measures are reliable and valid?
Specifically, you should specify some evidence that your measures
generate reasonable levels of alpha reliability or test-retest
reliability as well as convergent and discriminant validity in
populations that are relevant to your study (See also Properties of excellent measures).
Task 15. Have you selected measures that are not too susceptible to social desirability?
Some measures are invalid because the responses of participants align
with social expectations or desirability. For example, if asked the
extent to which they, "Cooperate with colleagues", most participants
recognise that such cooperation is socially desirable. Hence,
participants will overestimate the extent to which they exhibit this
behaviour. This bias presents complications only if both the
independent and independent variables are susceptible to social
desirability. In this instance, social desirability is tantamount to a
spurious variable (see Socially desirable responding).
Several options are available to prevent this shortcoming. First,
scales have been created that gauge the extent to which the responses
of individuals align with social desirability. In essence:
- Participants are asked to specify whether or not they have engaged in
some undesirable, but prevalent, act such as overdue library books.
- Participants that deny they have engaged in these acts are assumed to align with social expectations.
- This trait can then be included in analyses to control the effect of social desirability.
- Unfortunately, these measures are obviously not a pure measure of
alignment to social expectations, especially because this construct is
probably multifaceted.
Second, some measures have been demonstrated to be uncorrelated with
scales of social desirability. This finding suggests these measures are
less susceptible to social expectations.
Third, in some instances, the desirable response is unclear. For
example, if asked to specify the extent to which they, "Worry about the
health of your parents", participants are uncertain which response is
desirable. Agreement suggests they might be neurotic whereas
disagreement suggests they might be heartless.
Finally, participants are less likely to be swayed by social
expectations when they recognise their responses are entirely
anonymous. Nevertheless, anonymity might not completely eradicate this
bias. In summary, you should:
- Demonstrate the measures are not unduly susceptible to social desirability.
- For example, cite studies that reveal that such measures are not highly correlated with scales that gauge social desirability.
- Alternatively, argue the scale does not measure some ideal trait or construct.
- Finally, you could argue the responses are sufficiently anonymous to preclude social desirability.
Task
16. Have you considered the sensitivity of your measures to this
population, with particular reference to floor and ceiling effects?
Sometimes, most of the participants will receive the same score on some
measure. For example, participants might all specify the maximum score,
called a ceiling effect, or the minimum score, called a floor effect.
This limited variability reduces power and thus obscures significant
findings. Past studies that have applied these measures to similar
populations, or even pilot studies, should be considered to preclude
this problem.
References
Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand McNally.
Podsakoff, P. M., MacKenzie,
S. B., Lee, J., & Podsakoff, N. P. (2003). Common method biases in
behavioral research: A critical review of the literature and
recommended remedies. Journal of Applied Psychology, 88, 879-903.