1. Glad you stopped in for a visit!

    Please feel free to peruse around and if you have a minute make sure to sign up for an account! We're one of the largest, most active, and helpful communities you'll find for learning about mTurk & registered accounts automatically get access to tons of useful features on the site.

    Features for members include:

    • The ability to post messages - our community is incredibly helpful & happy to welcome newcomers who are looking to learn the ropes!
    • Live thread update - no need to refresh pages to see new posts/HITs they flow onto the page automatically as they're posted.
    • Desktop alerts - direct PANDAs from exported HITs right on your desktop
    • Access to the full list of HITs exported through the Daily HIT Log
    • TurkerView account which will grant you extra access to HIT review information all across mTurk through TVJS

    But most importantly you'll be joining an amazing community. You'll find all types here, from stay-at-home Moms to IT pros. We're a great group - feel free to join us!

GeoHIVEs Data Quality / Qualification / Wages Discussion

Discussion in 'Requesters' started by Bobby, Jun 8, 2018.

  1. James Wilson

    James Wilson GeoHIVE

    Messages:
    6
    Gender:
    Male
    Ratings:
    +17 / 0 / -0
    I appreciate you taking the time to lay out your argument. In response, I realize we weren't clear when we talked about people doing hits too fast. 95-99% of the people who participate in our campaigns do as you say; they go about their own pace and provide quality work. However, we take issue with people who complete a disproportionate number of the HITs at speeds that are beyond humanly possible and provide invalid responses. To us, that is gaming the system. We understand that scripting and shortcuts are beneficial to workers. We don't take issue with the use of scripts, so long as workers honestly complete the work.
     
  2. lefty

    lefty Well-Known Member

    Messages:
    15,159
    Gender:
    Female
    Ratings:
    +40,889 / 22 / -0
    I also haven't done a geohive in probably a year or so. As I learned more about the mturk platform I also learned where the better money is, and haven't really looked back to you guys. I also am in a bit of a different category as I have a full time job, so I'm even MORE picky when it comes to what hits I will do since I have a real income to fall back on.

    Your hits don't pay enough for me to spend time on, and I can only imagine that people who do pay attention to them are providing less than stellar data because of that.

    I wholeheartedly agree with everything ezdz said. The way this market works you're going to get the new people who don't know where better requesters are, and the people who provide substandard work to meet whatever hourly they feel is worth it to them. While I can't speak for everyone, but myself (and many people I am aware of) use at least one simple script per hit we work on, and that is a wage calculator. We do a hit or two to get a feel for the pay, and then either dive in or move on.

    My advice to you is have a few of your workers (different people) sit down and complete a batch of your hits and gauge their times (tell them to pretend this is their only income), and their mess ups, and come up with a possible few things:

    1) better pay scale
    2) a system for errors to not cause qual revokes from mistakes (unless you want to be draconian, but this will drive people away if you rejects and revoke with no communication)
    3) quality control questions
    4) revoking of a qual when people don't meet your quality threshold (I don't think the threshold should be absolute 0 mistakes ever)
    5) breaking up the hits, @Eisenpower mentioned an assembly line and it's spot on. You're going to get better overall data breaking up stuff and making it more linear per person.

    I am a great worker to my IRL job. I bust my ass and go above and beyond, but they also pay me higher than the minimum wage and I have lots of extra benefits. And if I happen to make a mistake, it's addressed appropriately and I am told what I've done, and I make sure to correct it and not cause it to happen again. Obviously if I didn't and kept repeating the mistake I would be fired and lose all this great stuff I have at this current job... so you can bet that if they say jump I'm asking how high and which way.

    I feel like sometimes people try to take the high road with this and be like "you should do your absolute best! Provide your best work ever!". I'm sorry, but for $10/hr you're getting teenager me and her attention span, not adult me with her experience and focus. Well to be frank for 10/hr you're not getting me at all because I rather pull an overtime shift at my IRL job, or work for the various 20/hr requesters that are up instead.

    I'm sorta just rambling at this point so I'll wrap it up. Those are my various two cents, and I hope it works out for y'all.
     
    • Love Love x 3
    • Like Like x 1
  3. tempo

    tempo Member

    Messages:
    70
    Gender:
    Male
    Ratings:
    +134 / 0 / -0
    @James Wilson I'd love to get a reply to my email about those 19 rejections. I'd like to receive an explanation and if possible some examples of my mistakes. I will definitely edit/remove my feedback from turkerview after that. Thank you!
     
    • Like Like x 1
  4. slothbear

    slothbear Well-Known Member

    Messages:
    6,975
    Ratings:
    +14,173 / 20 / -2
    @James Wilson - Thanks for getting in the discussion and providing info. This comment is aimed more at the .10 batch, but I guess could be useful for all the batches.

    I think examples of edge cases would be helpful. Maybe a link in the instructions to an example page instead of filling up the hit with examples. For example, on the .10 hit, I wasn't always sure when to use the industry tag. Take warehouses for example... I know that a huge warehouse with semi trucks and/or rails leading to it would count (example 1), but what about smaller structures that may be a small warehouse, but could just as well be a strip mall or other commercial type center (example 2)? I wasn't sure about mines counting as industry, but they weren't listed so I wasn't marking them. Or if the image is all grassland, but there is one house somewhere (example 3), is that still marked as residential? I was marking those because I think that's want you wanted based on the instructions.


    tldr: A few typical examples and then some edge case examples may go a long way to helping you get some better data.
     
    • Like Like x 1
    • Love Love x 1
  5. Eisenpower

    Eisenpower Well-Known Member

    Messages:
    4,697
    Gender:
    Male
    Ratings:
    +11,673 / 8 / -0
    That's why I mentioned the subjectivity of it. The responsibility is on us to make a judgment in those cases. All we can do is make our best judgment given the information we have. Everything you mentioned, minus mines, is what also went through my head when I did them.
     
    • Like Like x 2
  6. Moa711

    Moa711 New Member

    Messages:
    2
    Gender:
    Female
    Ratings:
    +1 / 0 / -0
    I agree with this, especially the fringe stuff. I have been marking am image with even 1 house as residential, even though that isn't residential

    Same for the industrial, if it looks like it could potentially be industrial I mark it, but that doesn't mean it is. It could just be a warehouse, green house, or any other large structure.

    Also in these latest hits, I can't tell the difference between the military base and an industrial park. The examples look interchangeable

    Also, if the image is a bunch of industrial buildings, do you want me to just toss down a bunch of markers?
     
  7. James Wilson

    James Wilson GeoHIVE

    Messages:
    6
    Gender:
    Male
    Ratings:
    +17 / 0 / -0
    In those recent batches, you did the right thing. If there was a single house and a single warehouse, we were looking for Residential and Industrial. We, GeoHIVE, need to do a better job of telling you our expectations and what to expect in the instructions. Several people have emailed us to ask for clarification and our expectations.

    In these batches, there are a lot of images that are hazy or dark and a lot of images that don't have any of the features present. Don't worry if there aren't any of the features present, and if the image is too poor quality, then tag Poor Image any where on the image.

    I agree, there's a lot of ambiguity between some industrial and military facilities. As the previous posts have said (@slothbear and @Eisenpower), a lot of this is subjective. Do the best you can and don't over think it.

    A single Industrial tag will work for an industrial area. No need to tag each facility. For this campaign there is really no need for more than 2-3 tags of each type.
     
    • Like Like x 1
    • Useful / Informative Useful / Informative x 1
  8. turker

    turker Well-Known Member

    Messages:
    1,549
    Ratings:
    +2,527 / 3 / -0
    Could you please tell us when the hits will be checked and hopefully approved for the last few days? The 2 batches that are up now.
     
  9. Momof1

    Momof1 Well-Known Member

    Messages:
    4,048
    Gender:
    Female
    Ratings:
    +4,267 / 0 / -0
    I would just like the request to do geohives batches to be approved. I submitted back in december of last year. :( Should I try again? (I so can someone give me a link)?
     
    • Like Like x 3