Showing posts with label validation. Show all posts
Showing posts with label validation. Show all posts

Saturday, February 25, 2012

Data Values Validation

Hello all...I am trying to validate that the new work date that is being imported from the text file does not exist in the table. So in other words I do not want duplicate data. This is what I haveSqlDataReader dr =newSqlDataReader(); SqlParameter sp =newSqlParameter("@.WorkDate", Data.SqlDbType.datetime, 8, Data.ParameterDirection.Input); if (dr.HasRows) sp ="@.WorkDate"; else *How does that look? Am I in the ball part…

Try using a stored procedure but instead of

INSERT INTO tablename (WorkDate) VALUES (@.WorkDate)

have

IF NOT EXISTS(SELECT * FROM tablename WHERE WorkDate =@.WorkDate)
INSERT INTO tablename (WorkDate) VALUES (@.WorkDate)

That way you can call the stored procedure passing in the WorkDate value, but a record will only be inserted if the value does not already exist.

|||

Thanks TAT for your help... question, what if a record in there with the same date how would I let the user know?

|||

To let the user know, you need an additional parameter so your stored procedure would look like this

SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
-- =============================================
-- Author: <Author,,Name>
-- Create date: <Create Date,,>
-- Description: Add record if not already on file
-- =============================================
CREATE PROCEDURE dbo.usp_TableNameAdd
@.WorkDate DateTime,
@.Exists BIT OUTPUT
AS
SET NOCOUNT ON;
SET @.Exists = 0
IF NOT EXISTS(SELECT * FROM tablename WHERE WorkDate = @.WorkDate)
INSERT INTO tablename (WorkDate) VALUES (@.WorkDate)
ELSE
SET @.Exists = 1
GO

|||

Another way could be to raise user friendly error message from within your SP. Read BOL for RAISERROR.

|||

I have a few more questions. When calling the sp I am getting an error message also in the SP the staring lines

SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO

are not working for me, is there something I am not seeing.

namespace WindowsApplication2
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}

private void button1_Click(object sender, EventArgs e)
{
//Pull from Text File in from Network Drive
string Source = @."\.\J-1119.txt";
string Date = "";
string O1 = "";
string A1;
string P1;
string R1;
string LineIn;
StreamReader InFile = new StreamReader(Source);

***TRYING TO CALL MY SP****
SqlCommand SC = new SqlCommand("SP_PromoNoDups", sConnect);

SC.Parameters.Add(new SqlParameter("@.WorkDate", WorkDate));
// SqlCommand SC = new SqlCommand("Select * from [tblname]", sConnect);
SC.CommandType = CommandType.StoredProcedure;

LineIn = InFile.ReadLine();
// pos is the location of the cursor
int pos = LineIn.IndexOf("WORK DATE:");
if (pos > 0)
{
Date = LineIn.Substring(pos + 10, 10).Trim();
}
while (LineIn != null)
{
if (LineIn.IndexOf("O1:") == 2)
{
if (LineIn.Length > 10)
{
O1 = LineIn.Substring(10).Trim();
}
else
{
O1 = "";
}
LineIn = InFile.ReadLine();
LineIn = InFile.ReadLine();
while (LineIn != null && LineIn.Trim() != "")
{
P1 = LineIn.Substring(2, 5).Trim();
A1 = LineIn.Substring(19, 20).Trim();
R1 = LineIn.Substring(41, 6).Trim();
string Adjustment = LineIn.Substring(57).Trim(); LineIn = InFile.ReadLine();
if (LineIn.IndexOf("mane") >= 0)
{
break;
} SC.CommandType = LineOut;(Erroring out here)
SC.ExecuteReader();
// if page header - look for next O1
}
LineIn = InFile.ReadLine();
LineIn = InFile.ReadLine();
}
InFile.Close();
}
}
}
}

Data Validation on merge agent

Hi,
I intend to set up data validation and re-initialize on merge agent. But
I'm not sure if it will cause data lost at subscriber side.
If the data validation fail, the merge agent is supposed to re-initialize
the subscriber from last successful data validation. Does it mean any data
change at the subscriber after the specific re-initialization point would
lost?
Please Help
Thanks
Yong
There is an option to upload the subscriber changes before the subscription
is reinitialized. If you use this option you will not lose any subscriber
data.
Hilary Cotter
Director of Text Mining and Database Strategy
RelevantNOISE.Com - Dedicated to mining blogs for business intelligence.
This posting is my own and doesn't necessarily represent RelevantNoise's
positions, strategies or opinions.
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com
"yong" <yong@.discussions.microsoft.com> wrote in message
news:8E312D5B-5A2D-494B-A6BE-62E056993DDB@.microsoft.com...
> Hi,
> I intend to set up data validation and re-initialize on merge agent. But
> I'm not sure if it will cause data lost at subscriber side.
> If the data validation fail, the merge agent is supposed to
> re-initialize
> the subscriber from last successful data validation. Does it mean any data
> change at the subscriber after the specific re-initialization point would
> lost?
> Please Help
> Thanks
> Yong
>

Friday, February 24, 2012

Data validation for datetime parameter in SSRS

Hi,

I wanted to know more about validation of SSRS parameters. I have a simple report which has a parameter called startdate of DateTime datatype. The datetime parameter in SSRS takes manual input as well. So, the user can enter any junk value. I want to ensure that the input parameter is in correct format and I want to display an error msg when the format is incorrect. My report has the following VB code for validation:

Public Function Validate( ByVal startdate As String) As Boolean
If IsDate(startdate) = True Then
Return True
Else
Return False
End If
End Function

And my report has a textbox which has the expression property set to;

=Code.Validate(Parameters!startdate.Value)

the textbox on the report has to display if the entered date is valid or not.

But, when i enter an erroneous date, SSRS doesn't render the report and throws a generic error. This happens even before the code written for validating the parameter executes.

Also couldn't find a way to disable the manual input for the datetime parameter. Even that would solve the problem.

Another alternative was to make the startdate parameter as string, but i want the calendar control button to be provided for the user.

The function below checks for the Start and End date ranges .

Function:

Function CheckDateParameters(StartDate as Date, EndDate as Date) as Integer
Dim msg as String
msg = ""
If (StartDate > EndDate) Then
msg="Start Date should not be later than End Date"
End If
If msg <> "" Then
MsgBox(msg, 16, "Report Validation")
Err.Raise(6,Report) 'Raise an overflow
End If
End Function

Steps:

1.) Go the Report Parameters and add a parameter with the datatype is string.

2.) Check the Hidden checkbox and Allow blank value ckeckbox.

3.) From Default Values choose Non-Queried radio button and then press the FX button and paste this code.

=CODE.CheckDateParameters(<parameterStartdate>.Value,<parameterEnddate>.Value)

Then press OK.

Hope this helps......|||

In the sample that you provided, if we enter an improper date as 02/31/2003 [mm/dd/yyyy], then the report execution fails and an error is thrown right away "error occured during the processing of report parameter". I was talking of handling such errors.

I feel that reporting services, initially validates the entered value matches the datatype of the parameter and then proceeds with execution of any VB code and finally renders the report.

When I said validation, I wanted a functionality similar to the client side validation in an asp page, where improper date formats like 12/31/235668 etc can be taken care of.

Data Validation Challenge

We have data migration process that transfers around 450 million records from
one DB to Another DB. There is a validation that takes place for the migrated
data that is time consuming (few days). The following validations takes place
on migrated data
NULL check
Length Check
Numeric Precision Check
So its looping through 450(rows) million * 30 (columns) times. so it takes
forever to complete the validation process and moreover the space
requirements also growing exponentially :-(. I would like to know is there a
better approach for validation of this kind. we are planning to try partition
approach. If there is any better way please help with your recommendations.
Regards,
Murali
Which tool are you using?
"Murali" wrote:

> We have data migration process that transfers around 450 million records from
> one DB to Another DB. There is a validation that takes place for the migrated
> data that is time consuming (few days). The following validations takes place
> on migrated data
> NULL check
> Length Check
> Numeric Precision Check
> So its looping through 450(rows) million * 30 (columns) times. so it takes
> forever to complete the validation process and moreover the space
> requirements also growing exponentially :-(. I would like to know is there a
> better approach for validation of this kind. we are planning to try partition
> approach. If there is any better way please help with your recommendations.
> Regards,
> Murali
>

Data Validation Challenge

We have data migration process that transfers around 450 million records fro
m
one DB to Another DB. There is a validation that takes place for the migrate
d
data that is time consuming (few days). The following validations takes plac
e
on migrated data
NULL check
Length Check
Numeric Precision Check
So its looping through 450(rows) million * 30 (columns) times. so it takes
forever to complete the validation process and moreover the space
requirements also growing exponentially :-(. I would like to know is there a
better approach for validation of this kind. we are planning to try partitio
n
approach. If there is any better way please help with your recommendations.
Regards,
MuraliWhich tool are you using?
"Murali" wrote:

> We have data migration process that transfers around 450 million records f
rom
> one DB to Another DB. There is a validation that takes place for the migra
ted
> data that is time consuming (few days). The following validations takes pl
ace
> on migrated data
> NULL check
> Length Check
> Numeric Precision Check
> So its looping through 450(rows) million * 30 (columns) times. so it takes
> forever to complete the validation process and moreover the space
> requirements also growing exponentially :-(. I would like to know is there
a
> better approach for validation of this kind. we are planning to try partit
ion
> approach. If there is any better way please help with your recommendations
.
> Regards,
> Murali
>

Data Validation

Being relatively new to SSIS, I'm looking for advice, or a best practice, regarding data validation before extracting the data for a transformation.

One of my project's require that certain data be validated in staging tables before it is loaded. The validations include checking for null values, verifying that a field is populated with apropriate values etc... The entire batch of data (good records and bad records) may be rejected depending on the validations.

I have a couple of different thoughts on how this could be handled...

    Run a series of validation queries on the data before executing an SSIS package Run some kind of validation transformation (does one exist or should I write a custom transformation?) Place contraints on the target tables so that bad records error out on the load Something else... I could be missing the completely obvious

#3 doesn't seem to viable as the entire load may be rejected if some of the data is bad...

Any thoughts?

You could filter out the bad data very easily using a Conditional Split transform. Better still, you can pipe that bad data somewhere for later analysis.

Does that help?

-Jamie

|||I like both of thoses suggestions. With the conditional split transform, could I then update the original record with a status flag?|||

Once you pipe the data off elsewhere from the Conditional Split you can do what you want with it so yes, you should be able to do this! You would use an OLE DB Command transform.

You may have a problem (i.e. blocking) when trying to update a table that you are selecting from in the source - but this can be easily alleviated by dropping the update dataset into a raw file and then doing the update in a seperate data-flow.

-Jamie

|||

Thank you!

I was just testing an update using the OLE DB Command Transform. The raw file seems like good work around if the row is locked!

Data type validation

Hi all,

I want to make a conditional split based on the data type provided by the input.

For example : If the comming (Column x) is of data type (numeric) then pass , else do not pass.

(pass = Case 1

Do not pass = Case 2).

Is there any way for doing so ?

You can use conditional split transformation for this.

The following link will help you get started
http://technet.microsoft.com/en-us/library/ms137886.aspx


Thanks

|||

I think Data Convertion transform is the easiest way to do this, see this for details:

http://technet.microsoft.com/en-us/library/ms186792.aspx

You would read the data as string, then try to convert it to numeric. Configure Error Disposition to redirect row. The "pass" will go to default (green) output, the rows that do not pass will go to error output.